Recently in Business Category

Perl Success Story: Client-Side Collection and Reporting

Accurate software inventory management is critical to any organization. Without an accurate software inventory, organizations may either be out of compliance with their vendor licensing agreements or they may be paying extra for licenses that they do not need.

Hitachi Global Storage Technologies (Hitachi GST) formed in 2003 as a result of the strategic combination of Hitachi and IBM's storage technology businesses. The company offers customers worldwide a comprehensive range of hard disk drives for desktop computers, high-performance servers, and mobile devices. With offices and manufacturing facilities spanning the globe, deployment of the corporate business intelligence (BI) tool suite is extensive. The company uses BI tools throughout the business, from manufacturing to sales and warranty operations, for analysis of data that reside in operational data stores and data warehouses. With each new manufacturing site comes requests for additional licenses.

A Business Intelligence Problem

To avoid unnecessary spending on licenses, the company needs a detailed assessment of both the number and type of licenses actually deployed at the various sites. We operate in a highly competitive environment in the technology industry and can ill afford to continue purchasing unnecessary licenses. An effort was started to determine how we deploy the current pool of licenses. The initial effort to perform a software inventory for the BI tool involved a manual inventory by each site owner or application owner. As expected, the results returned appear to indicate a significant under-licensing with one of the products from the BI suite. Looking at the raw information sent in by each owner, we had speculated that we double-counted some users and that some users may be misreporting the client that they have installed on their system, thus driving up the license count for the respective product.

Based on the preliminary analysis of the information collected, it became evident that we needed to deploy a software inventory tool to assess each workstation and determine what users have actually installed on their machines. The corporate software asset management tool was not a viable option, because it had limited deployment, with some organizations maintaining software inventory using manual methods. The licensing issues of deploying a commercial tool, along with the customizations needed in order to accurately capture the different types of licenses, would be prohibitive from both a time and cost perspective.

In order to generate an accurate license count, the software inventory tool must be able to detect the different types of BI clients that may be installed on a given PC. The release information of each version is also important. We can use this information to determine which users need upgrades to the latest supported release. Our license entitlement limits the number of each type of client from the BI suite that we can deploy across the enterprise. To further complicate matters, there are two different types of client-server tools that use the same binary, with no easy way to distinguish between them by looking at the installed applications listing in Windows. The difference in cost between these two tools is approximately $3,400 per license, so it is imperative that the inventory count is correct for each specific type of client. In addition, a web-based version of the BI tool is available as a plugin. However, the plugin version information is inaccurate in the binary file and the client-server tool version is incomplete in the registry. In preliminary tests of a commercial tool, it was unable to distinguish between the two types of clients and missed the web plugin altogether.

The Design of a Solution

Given this challenge, we had to develop a custom solution to address the problem. I had previously used Perl to develop an automated software delivery and installation solution on the Windows platform, and some of what I had already developed was very similar to what we needed. Leveraging this prior experience reduced the amount of time and effort needed for development of the software inventory solution to address our immediate needs.

The solution criteria that we set forth required that the client inventory tool must be relatively compact and not require any installation. The tool also must be a self-contained executable that a user can download and run on their client. If a tool is difficult to use or install, this severely curtails its adoption and use. The user population is spread across multiple locales, including Japan, South Korea, Taiwan, Singapore, Thailand, the United Kingdom, and the United States, so the tool must be simple to understand and must avoid complex procedures.

Each user also needs to authenticate against the corporate LDAP directory to ensure that their results are accurately recorded and ensure the authenticity of the result. In addition, the inventory results must be collected and stored on a central server to ease management of the inventory process. The results must also be readily accessible to all users and owners so they can see where they stand in comparison to what they had originally reported.

The solution involves a client tool and a server-side application. The combination of the two must authenticate the user against the corporate enterprise directory, inventory the client system, log the results, and record the results in a DB2 back-end database. The data stored in the DB2 back-end database is useful for generating both summary and detail reports via the Web. The specific modules used in the development of the solution included Win32::OLE, Win32::GUI, Win32::File::VersionInfo, Win32::TieRegistry, IO::Socket, CGI, DBI, DBD::DB2, and Net::LDAP.

Building the Client

Development started with the client component first. Because most end users are averse to command-line tools, we had to develop a GUI of sorts to guide the user through the authentication and inventory steps. In addition, the GUI could display error messages to the user. We used Win32::GUI to create the prompts as well as data entry dialog boxes. The GUI captures the information entered and passes it to the central server using HTTP.

All communications between the client and server use HTTP because coding the client to work directly with the LDAP server and the database server would have made troubleshooting any problems experienced by clients in distant locations inordinately difficult. In a situation such as this, we wanted a single controlled point of failure, as depicted in Figure 1. By using HTTP and a central-server-based application to broker requests, we can isolate problems on the client side to only HTTP-related transport issues.

The authentication process
Figure 1. The authentication process

The user that is running the inventory tool must authenticate first. This is important, as we need to track who has or has not reported their results based on the initial manual inventory information. We use Net::LDAP to perform the authentication against our LDAP server. The application prompts users for their intranet IDs and passwords, and the client passes this information to the server application so that it can issue a bind against the LDAP server. If the information provided by the user does not yield a successful bind, then the client requests that the user reenter his or her login information. Figure 1 depicts this authentication process.

To authenticate a user via LDAP using Net::LDAP, the server uses the user's intranet ID to prepare the distinguished name (DN). The DN is the unique identifier for the user's record in the LDAP directory. In order to perform the initial lookup, the server issues an anonymous bind against the LDAP directory and performs a search using the intranet ID provided. The search returns only the uid, which is the unique record identifier in our implementation of the corporate directory. After the server obtains this, it forms a DN string and issues another bind again using the password provided. If the bind is successful, the return code is undefined and the client proceeds to inventory the system.

Most software applications leave some type of signature in the system registry on a client system, and the BI tool suite deployed at Hitachi GST is no exception. I knew that the software leaves specific signatures when installed on a client system. This information is in the system registry under both the HKEY_LOCAL_MACHINE and HKEY_CURRENT_USER registry entries. We use Win32::TieRegistry to retrieve registry values for determining the type of client that is installed, as well as the binary install location and install key used. Win32::TieRegistry makes it extremely easy to access and modify registry values.

The exact version of the BI client deployed is not accurate in the system registry as we also want to determine the specific point release deployed on the client systems. To achieve this, we use Win32::File::VersionInfo to extract the ProductVersion information from the binary files. Looking at the properties for the file in Windows Explorer for the BI web client binary, we can clearly see that the FileVersion information in the executable is incorrect. We settled on using ProductVersion instead.

In addition to checking the various client versions that may be installed, the tool must also log the client machine serial number. We achieved this by using Win32::OLE and querying information from the Win32_BIOS WMI class. Because some users may have multiple workstations or laptops, we use the machine serial to differentiate multiple entries logged by a single user. The WMI classes yield a wealth of information. Extending the features of this tool simply requires querying additional WMI classes to obtain additional hardware information, such as hard disk, CPU, and memory.

Deploying the Client

Once we developed and tested the client code, we had to package the script itself. Not every client machine has Perl installed. Even if they did have Perl installed, there is no guarantee that the appropriate modules needed by the application are available on the target systems. To package the final script into a single executable file, we used PerlApp from ActiveState's PDK to compile the script and the associated modules. This allowed us to neatly package and deploy the client as a single executable that users can run from their client system without performing any type of installation or setup.

When all of the requisite information is collected on the client machine, it passes back to the server which records it on the server's local file system first. This initial logging allows us to capture the client inventory results even if the database server is down. The logged results are easy to import into the database, should there be an unexpected database outage. The CGI module handles the information passed from the client to the server, and DBI and DBD::DB2 handle the database connection. DB2 is the RDBMS of choice in our environment, and we were able to leverage an existing DB2 database environment, which further reduced the time to deployment.

Learning from the Process

All of this information that we collect is of little use if it is not easily accessible for analysis. We also wrote a script to retrieve and format information from the database and present it via the Web. This allowed users and application owners to see the current inventory status of their users, and also help them to determine which users still need to run the client inventory tool to complete their assessment. The script provides summary, detail, and exception views. With reports available through the Web, users can access it from any web browser, regardless of the user's geographic location. This empowers everyone involved to see the results in near real time.

Upon deployment of the tool, we then recorded and summarized user inventory information automatically. Some interesting results came to light. It became apparent that up to $163,000 worth of additional license purchases were unnecessary, as the existing pool of licenses had not been depleted (as we believed earlier), based on the manual inventory results submitted. Also, with the client inventory tool being an internally developed tool, there were no licensing costs for distributing it at various sites.

The client- and server-side components took two weeks to develop. We formed test cases before writing the code in order to minimize functional issues during final testing in the field, as well as to ensure that we did not overlook critical features and integration issues during the development process. Ultimately, as illustrated in Figure 2, the client and the various server components must work together as an integrated system.

The client and server working together
Figure 2. The client and server working together

The use of virtual machine software was also instrumental for testing and reduced overall development time. It allowed me to simulate client environments and test client code without having to use a physical machine for each version of the BI application. Using undo disks, I was able to return the VMs to a prior state very quickly without manually uninstalling the various BI tools from the VM. We initially tested the inventory tool in Singapore, Japan, and the U.S., and this testing confirmed that the tool was able to perform well even though users were spread across remote geographic locations.

Although we developed this tool for a one-time inventory of the number of BI clients deployed in the field, it also has a role going forward in helping to maintain an accurate picture of the licenses deployed. Possible enhancements include converting the tool so that we can install it as a service to run without user intervention for periodic updates of client inventory data from identified clients, auto-conversion of licensing from one client class to another, and automatic uninstall of clients, should the user decide that he or she no longer needs it. In staying with the near-zero client management paradigm, we can extend the client to contain auto-update features by checking the local version and verifying it against the latest published version on a central server.

How Perl Powers the Squeezebox

Slim Devices made their name in 2001 with the SLIMP3, a networked MP3 and Internet radio player. The SLIMP3 won a five-out-of-five mouse rating in Macworld magazine, and was featured in GQ magazine and on TechTV's The Screen Savers. This year, they're trying to repeat the success with Squeezebox, an 802.11-enabled version.

The interesting thing about the SLIMP3 and the Squeezebox from our point of view, though, is that the server which drives the player is written in Perl, and developers are allowed -- in fact, encouraged -- to hack on it and make the devices do interesting things. A developer community has sprung up around the SLIMP3, and some interesting third-party hacks have been produced.

We caught up with Dean Blackketter from Slim Devices, who took some time out from moving Squeezeboxes to answer a few of our questions.


perl.com: What made you decide, first of all, to write the server code in Perl?

Dean: Sean [Adams, Slim Devices founder] wrote the very first version of the software in Perl because he was able to get it up and running quickly. When I first encountered Sean and SLIMP3, I was a little afraid of Perl, having been primarily a C and C++ programmer, mostly because when I tried to read it, I got confused by idiomatic usage.

I first decided to rewrite the thing in C, but I couldn't really get started until I could read the existing code. So I bought a copy of Learning Perl and read it in two evenings. Halfway through the first chapter something clicked and I said, "That's so cool!" Then, about every 10 pages after I'd repeat, "That's so cool!" again, but a little louder. By the end, I was sold. At that point I dug in and started making the software better.

What features of Perl were particularly helpful when you were writing the SlimServer?

The greatest feature of Perl, for us, is that so many people know it. Since our software is open source, we benefit from as many users as possible being able to read and improve our code base. Anybody who's thrown together a little Perl CGI or script is a potential contributor to our software.

Second, the ability for us to offer our software on a wide array of platforms (Mac OS X, Windows 98/ME/NT/2000/XP, BSD, Linux, Solaris) and also be deployed on server appliances makes it possible for us to get our product running as broadly as possible.

Third, even though our product's primary feature is playing back audio, the bulk of the software really is processing text. Our built-in HTTP server, command-line interface, device-user interface, and even music meta-information is all text-based, and Perl is nearly the perfect language for this.

What issues did you have to face when considering how to distribute a Perl-based consumer application, and how did you approach them?

Distributing a Perl application on the traditional Unix-like platforms (BSD, Solaris, Linux) is easy. Things got a bit harder on Mac OS X, since command-line solutions aren't acceptable to Mac users. Initially, I wrote a simple graphical application (in AppleScript!) that acted as a launcher for our server software, but this was really not good enough for Mac users. One of our Open Source contributors, Dave Nanian, came to the rescue and built us a System Preferences pane and installer application that work as they should on a Mac.

Windows is the hardest platform for us to support. Luckily we had another Open Source contributor, Sam Saffron, who decided that building a shell application for the server software would be a great way to learn MFC, and he contributed a nice little app for this. Another contributor, Chris Eastwood, put together an installer for us. But it was the fine folks at ActiveState with their Perl Dev Kit that really made it possible for us to release on Windows with their PerlApp (for making Perl-based .exe files) and PerlSvc (for making Perl-based Windows Services).

Even with all this help, Perl on Windows is different enough in subtle ways to make it a real pain to maintain. Maybe if we had more hardcore in-house Windows experience things would be easier, but it's a struggle.

For consumer devices like the SLIMP3, the manufacturer has a choice between open and hackable, and "sealed box." You guys went for the hackable approach, GPLing the code and setting up developers' forums. What was the rationale behind that?

When I got into this, the total of my Open Source experience was running a Linux server in my basement. Sean initially released the SLIMP3 software under the GPL, which made it possible for me to help out. I quickly realized that our customers have a lot of talent and energy that they are willing to contribute to make the product better. We're lucky to have a business model where we can make money on the hardware and give the software away. When our customers contribute, they make their own SLIMP3s (and now Squeezeboxes) more valuable to themselves and make the new units we sell to new customers more valuable as well. Everybody wins.

How did you go about building a developer community behind the SlimServer project?

I like to think that the community built itself. We provided a forum (our discussion lists) and Sean and I participate on a daily basis; acting on the requests, suggestions, bug reports, and patches that people post. We're spending more effort lately setting up our own list server and CVS system, as we had problems with our previous forums (Yahoo Groups and SourceForge, respectively), but these will give us a little more control of our virtual spaces.

How do you decide which developer extensions to keep as third-party projects and which to integrate?

If a contribution will make the product better for a substantial portion of our customers and won't make the product worse for any of them, then, generally speaking, it's included. Submitted features that make the software harder or less intuitive to use, require obscure or platform-specific software or hardware to be installed, generally don't make it in. Contributions that are useful to a large fraction of our customers and don't diminish the product's ability to play back music, make it in.

I act as final arbiter and (hopefully) benevolent dictator, and the community has been really supportive of this.

Finally, we notice that you've decided to donate 10 percent of your profits to the EFF. What's the motivation behind this?

Digital freedoms are important to the folks at Slim Devices, both personally and at a corporate level. We've been frustrated by not being able to offer our customers the ability to play the music that they've bought due to digital rights management systems and onerous licensing fees for various patents. The EFF shares our goal to free our customers' music.

The Squeezebox is available from Slim Devices and their resellers, and the server code can be downloaded from their downloads site.

Radiator

Are you fed up with those who think that commercial applications need to be written in an "enterprise" language such as Java or C++? While we're big fans of open source at Perl.com, we're bigger fans of Perl, and we're frustrated when people claim there's something that Perl can't do; so we spoke to Mike McCauley at Open System Consultants.

Open Systems produces Radiator, a commercial RADIUS server implementation, first released four years ago and with new versions and enhancements continually developed. It has about 5,000 paying installations worldwide. Mike explains: "It's used to authenticate dialup and wireless access to an ISP or corporate network. In order to authenticate users, it can look up user details in a wide range of data sources, such as SQL, LDAP, flat files, Unix password files, OPIE, PAM, Windows NT SAM, Active Directory, third-party billing packages, and so on."

Mike was particularly impressed with Perl's write-once-run-anywhere nature, which has completely obviated the need for porting or platform-specific alteration. As he says, "The finished product runs without change on almost every platform known to humanity. That means we can appeal to more potential customers, with less effort spent on porting and maintenance."

But there were other reasons for choosing Perl as an implementation language: "The richness of Perl allows us to express complicated algorithms quickly and concisely, and know they will work wherever the customer wants to run the code; and the connect easily to lots of different data sources." Perl's reusability and the great library of code already available also played its part. "The modules from CPAN mean we can talk to servers like SQL, LDAP, NISPLUS, Active Directory, OpenSSL, and lots of other things. We can concentrate on writing the product, rather than coding and maintaining interfaces."

We asked about the performance of going with Perl instead of, for example, C, but this seemed not to be the problem that many people might expect. "Most authentications rely on some external server or system, such as an LDAP or SQL server, and so the speed determining step is usually that extenal system. Also, the interfaces to those systems are generally compiled Perl modules written in C. And where there is no external server, we can use clever hashing mechanisms to make lookups faster than some C based Radius servers. That means that Radiator can be up there with a server written in C, and with much more features and flexibility."

Mike also explained that good Perl programming practice can keep performance high: "In order to get the best performance, you have to code so that you use as much of Perl's internal lovingly hand-wrought C code as possible. That means using complex operators like map, grep, hashes etc, to do the maximum of work with as few Perl operators as possible. You have to balance that against readability though, otherwise you can end up with unmaintainable code that looks like line noise. Radiator makes heavy use of Perl's Object Oriented support, which costs in performance, but we think the benefits in maintainability and easy extensibility (for us and the customer) are worth it."

One common concern with businesses releasing Perl products is that they're worried about piracy; if the source code is visible by anyone, isn't it easy for people to run away with it? The evaluation version of Radiator is shipped with a small portion of the code encrypted, and only made available to bona fide equirers. However, the full product is shipped completely unencrypted. Mike flips over the concern and sees the advantages. "Most network operators really like the idea of a product with full source code: They can be sure that the product does what it claims, and they can change or enhance it if necessary. ... We like to offer full source code, but we also need to be paid for our hard work, and partial encryption of demos seems to be a good compromise that results in most of our demos turning into sales."

What, then, about people making customizations or passing on copies to their friends? "Actually, we don't mind if customers change their code to suit their own needs. But we don't like it if they give the code to someone else, so that we get to support them without being paid. There is a little bit of that goes on, but ours is not really a mass-market consumer product, and I think that most people realise they get more benefit from an honest relationship with us. In the end, a license is not very expensive, especially since the support is so good."

Finally, we asked Mike to sum up his thoughts on commercial development with Perl. "Technically, I think it is unsurpassed for almost any application. In Perl, I can be five to 10 times more productive, line-for-line than in C or C++ (and I'm no slouch at them, either). For software vendors, that means a more maintainable product delivered to market faster. For customers it means a better product for less money. And interoperability and portability is fantastic. As you can tell, we really like it!"

"Commercially, a qualified yes, provided you have control over licensing and distribution issues, which might be hard in anything other than a niche market. You can't keep writing software unless you get to pay the mortgage and feed the kids, too."

Mike McCauley is chief software developer at Open System Consultants in Melbourne, Australia. He has a bachelor of engineering from the University of Queensland, and has worked in the computer software industry for 20 years. When he is not writing software, he flies light planes and has fun with his family.

Visit the home of the Perl programming language: Perl.org

Sponsored by

Powered by Movable Type 5.02