Ratchet-X and ETL

We recently exhibited Ratchet-X at Microsoft’s TechEd 2009 show in Los Angeles California. Thank you to all who visited our booth and made the show such a big success for us. While there was no shortage of ideas as to how Ratchet-X could be used within the organizations of those who took the time to receive a demo, there was one repeated use case that stood out. Let’s just say that despite all efforts the industry has made to create better integrated systems, ETL (Extract, Transform and Load), is alive and well. We heard a number of stories about how organizations still continue to struggle with creating and maintaining reliable ETL processes.

No doubt, modern application development and deployment technologies coupled with “smart” infrastructure are enabling organizations to integrate more-and-more systems in real time. However, despite the advances, ETL is still a reality of life in organizations of all sizes. So the question asked often asked of us at the show was; “Where does Ratchet-X fit within the ETL process?” Glad you asked.

Let me first say that while Ratchet-X does extract, transform and load data between systems, it is not a traditional ETL tool. When I think about ETL, I think of scheduled batch jobs that process and move data between queues and systems. Generally, ETL processes are executed during non-prime time usage or maintenance hours for the purpose synching data across systems. While the term ETL can be used to describe other related processes, I think this is what most folks mean when they refer to ETL.

So, if Ratchet-X extracts, transforms and loads data, why isn’t it an ETL tool? It’s mostly a matter of perspective. Whereas most ETL processes are scheduled to run in batch, work with large data sets and have little end-user interaction, Ratchet-X is an on demand data processing platform that processes a single record set at a time and is often guided by the end-user.

Most of our customers use Ratchet-X to add new features to existing applications without changing those applications in any way. These new features usually involve:

• Acquiring data from an external source (e.g. web services, web site, electronic form, database, API, etc).
• Sending application screen data to an external source.
• Processing application screen data for the purposes of either transforming it in some way or kicking off a down stream process.

In most cases, Ratchet-X is processing one or more application screen’s worth of data that collectively constitutes a single record set (i.e. a composite customer record, an insurance policy, a shipping manifest, etc.). Ratchet-X is not usually used to process thousands of these record sets at a time. Further, Ratchet-X is almost always used under the watchful eye of the end-user. In reality, Ratchet-X is an “assistive” technology that allows an end-user to acquire, push or process data to and from the systems they use without having to rekey data and reducing errors. This doesn’t sound like ETL to me.

So, uh, where is it that Ratchet-X fits into the ETL process? Oh yeah. No ETL process is perfect. In fact most ETL processes have an error handling stage where exceptions and records that fail validation are kicked out to an exceptions queue. Once in the queue, the user then must manually review and rekey the corrected information into the target system or error handling interface. This is precisely where Ratchet-X can be very useful. For example, we have customers that set up Ratchet-X’s task list as the exception handling queue so users can easily review the data, make whatever corrections are necessary and immediately paste the corrected record into the target application.

So while Ratchet-X is not an ETL tool per se, it can serve as an important part of the error handling stage of most ETL implementations. If you have any questions or comments regarding how Ratchet-X can be used to supplement your ETL process, please drop us a line. We’d love to hear from you.

If you'd like to see a video regarding how Ratchet-X can be used as part of your ETL efforts, click here.

SOA Dead...Again?

The Death of SOA…again? How many times have I read this headline? I thought it died back in 2006 when it was mortally wounded and replaced by Web 2.0…which then, according to the pundits, itself perished in Spring 2008. It’s so hard to keep this stuff straight. The good news is you don’t really have to.

While the media is replete with proclamations about the state of technology movements and buzzwords, discussions such as these are merely academic. Why is this? It has a lot to do with the media’s lack of nuance. To those who opine on such matters, attention spans are short so often something is either in or out, big or small, paramount or irrelevant. Fortunately, that’s not the way the real world works.

I agree with Burton Analyst Anne Thomas Manes’ summation that services is where it’s at. I’ve always held that belief. I was highly suspicious when discussions of web services gave way to SOA because the conversation transitioned from a compact and tangible concept to a much larger and nebulous concept. Read achievable versus hard to achieve.

When companies turned their attention to the broader idea of SOA, the appeal of project-based implementations and small wins went out the window. SOA became a religion that organizations had to strategically adopt from the start otherwise its benefits were not to be had. I strongly disagree. While most organizations would love to strategically embrace SOA from day one and experience “spectacular gains”, in most cases it’s not a practical approach. Small implementations, lessons learned, quick wins and replication is a much more sensible path. Over time, as success builds upon success, services ultimately become the way business functions are exposed by an organization. Along with it come the registries, management and scaling layers required for broader success. Dare I say, these services and the methodologies used to design, develop and deploy them ultimately become strategic? I'll buy that.

So if we’re bidding the term SOA a fond adieu, so be it. What’s in a name? But let’s go easy on the “SOA is Dead” and “…great failed experiment” language. The more these kinds of terms are thrown around, the more they stick with decision makers and are used against those fighting the good fight to implement services.

Quick Copy: Simple Elegance

The more full-featured Ratchet-X becomes, the more tempted we are to unduly emphasize the product's more “cool and complicated” features during demos. Now I’m not saying these features are not useful. No doubt they are or we wouldn’t have funded their development in the first place. However, this emphasis on the WOW features sometimes causes us to gloss over some of the other more mundane, yet equally as valuable features.

For example, have you ever used Ratchet-X to move data from one screen to another? While we spend a lot of time talking about integrating existing application screens with external data sources, we tend to down play the simple screen-to-screen copy feature. That’s unfortunate because virtually every Ratchet-X user indentifies this as a need, and yet we tend to demo this feature only after being asked.

So, for all of you who constantly enter data into multiple screens either for data entry purposes or launching queries in other desktop systems, I recommend you give the Quick Copy feature a look. Quick Copy is a simple feature that allows you to link fields on multiple screens so that data can flow between these screens in either direction. Simple, but pretty powerful when it comes to speeind up those pesky repetitive typing tasks.

Check Quick Copy when you get a chance. Better yet, give me a call and I’ll demo it for you.

Ratchet-X and Application Rationalization

Over the past two years or so, a number of consulting organizations have added “Application Rationalization” to their roster of services. Simply stated, application rationalization is the process by which an organization assesses its portfolio of application software and decides which applications need to be maintained, sunset or replaced. While this process may look different from organization-to-organization, the common goal is to reduce application bloat and functional redundancy thereby reducing overall IT cost.

Ratchet-X can be a very helpful tool during the application rationalization process. Often, applications are kept in production because they provide some functionality that can’t be added to a preferred system. Ratchet-X allows organizations to add or integrate that functionality into the preferred system quickly and easily thus allowing the redundant system to be sunset. Further, the rationalization process uncovers a slew of integration code and cron jobs that shuttles data between systems. Ratchet-X can help alleviate this web of spaghetti code by allowing organizations to deliver information normally moved in batch, to applications in real time at the point of need.

So, if you’re engaged in or considering undertaking an application rationalization project, keep Ratchet-X in mind as you evaluate system consolidation.

Document Enable Any Application With Ratchet-X

I recently received an email from a customer saying; “I’m amazed at how many places within our organization we’ve applied Ratchet-X beyond the original project.” We hear this all the time. Although I often make this point during customer presentations, it usually doesn’t set in until the customer lives and uses Ratchet-X for a bit.

One example where Ratchet-X is repeatedly applied beyond the original customer project is in the area of enterprise document management (EDM). Once customers realize that Ratchet-X is actually a piece of desktop middleware that brokers a “conversation” between application screen data and external data sources (and not merely an integration tool purchased to solve a specific problem), integrating applications into the customer’s EDM system is a natural fit.

Ratchet-X is now commonly being used as a way to both submit documents to, and retrieve data from, EDM repositories. On the capture side, Ratchet-X works best with images and forms since the information used for indexing the documents is often available in the integrated application screen. With regard to document retrieval, pretty much every application user can benefit by an onboard ability to instantly link to documents that relate to the entity referenced by the application screen data. Since Ratchet-X enables more applications, and by extension users, to leverage the EDM, the costs associated with purchasing and maintaining the EDM is justified over a much larger pool of users.

Not many products can claim they dramatically impact the ROI of the other products running within the organization. So try integrating Ratchet-X with your EDM back-end and let me know how it goes. Also be on the lookout for the release of Ratchet-X plugins that integrate into the industry’s most popular EDM products. If you’re thinking of doing such an integration, make sure you check with us to see if we have a plug-in for your EDM in the pipeline.

Twitter - I Just Don't Get It! Never Did, Never Will.

I know this has little to do with the purpose of this blog but I can't take it anymore. Today, I had yet another associate of mine invite me to track his every movement on Twitter. This comes just one day after telling another colleague that I refused to communicate with him exclusively through Pownce.

Does anyone over the age of 18 really need to know at this moment which friend is picking the nuts out of his teeth from the Snickers bar he just ate? The only reason I know it was a Snickers bar is because I received an urgent alert to that fact five minutes prior as he was unwrapping said Snickers bar.

Just because we have the technical ability to transmit every fleeting thought and the undertaking of every mundane task doesn't mean we should. But what do I know, I'm just a cranky old geezer with a little less time on my hands than some.

If you're connected to me via Twitter, you probably know that already.

What's New in Ratchet-X Version 3.5?

Good news for RatchetSoft users! Ratchet-X version 3.5 was released to the Ratchet-X Community last week. Here’s a review of what’s new.

Assembles
The most significant change in version 3.5 is the introduction of Assembles. While Ratchet-X has always had the ability to extract, paste and manage datasets that scan multiple applications and application screens, version 3.5 shifts the responsibility of supporting this feature from the plug-in developer to the appspace creator and the platform itself. As part of our ongoing effort to focus the platform on managing desktop data rather than mere application automation, we felt that multiple application and screen support should be something that the plug-in creator inherits rather than have to specifically plan for. This supports the philosophy of loosely coupling applications and data sources.

So now, if a plug-in requires data from an xmodel that spans multiple screens, it’s up to the appspace developer (the only person who really knows this to be the case for a particular application integration), to define that fact by instructing the Commander to create an assemble in the task list when the user executes the associated plug-in. Once the assemble is in the task list, it becomes much easier to be repurposed to other applications and sources. For more on assembles, check out the Ratchet-X documentation.

Macro Throttling
Appspace macros are created by the appspace author. This being the case, defined macro execution speed is usually based on the performance the appspace creator sees on his/her desktop. However, once the appspace is deployed, the macro’s execution speed may need to be altered to accommodate the performance of an end user’s workstation. This is the purpose for macro throttling. Macro throttling allows each individual end user to globally speed up or slow down the performance of macros executed on his or her desktop (five levels of throttling).

AutoSave/AutoBackup
In version 3.5, the Appspace Editor has both an AutoSave and AutoBackup feature. When Appspace Editor saves, it makes a backup of the previous version. Further, once you save an appspace for the first time, Appspace Editor will autosave a copy of your appspace based on a user defined autosave time interval.

In addition, there have been various changes to the the connector API and plug-in API to accommodate changes to the task list and assembles.

If you have any questions or comments regarding the new version, please gives us a call or drop me an email. Good luck with version 3.5 of Ratchet-X!

The Evolving Role of Organizational IT

I recently read two blog postings that touched on the future of organizational IT with regard to the rollout of new technology. The first entry encapsulated the view of venture capitalist Kevin Efrusy of Accel Partners. Kevin’s take is he is increasingly seeing technology sold directly to end users – the folks who pay for and receive the benefit from new technology. In fact, he’s putting his money where his mouth is by funding companies that deliver solutions users can find, buy and implement (or at least pilot), on their own.

The second posting was from Jeremy Geelan covering an interview he did with Rod Smith from IBM. Based on the posting title, the posting was supposed to be about IBM’s SMASH (secured mashups) project. However, the focus quickly switched to the concept of user empowerment. The idea being that organizations have pockets of non-technical users who are constantly looking for new technologies that can help them become more productive. He calls this group “shadow IT”. Cool name. No doubt, the concept of mashup and situational applications plays directly into the hands of these groups.

I strongly agree with both gentlemen’s sentiments. In fact, we see it ourselves everyday. While we spend a lot of time speaking with IT folks about how Ratchet-X can help an organization integrate the unintegratable, increasingly, we are brought into organizations by the end users who will directly benefit from a Ratchet-X implementation. Overall, I think greater user involvement in finding, evaluating and even piloting technologies is a good think. However, where there is opportunity, there is danger. While these user groups have the best of intentions in mind when seeking a solution, their views are often myopic and skewed towards their specific needs. This is totally understandable. Salesman are looking for tools that help them sell more, faster and better. They’re not in the business of worrying about what the folks in Accounting need. Nor should they be, that’s not their job. However, if they are bringing new technology into an organization, someone has to think about scale, integration, support and maintenance. This IT’s job.

IT’s role is changing in that in the past, requirements were developed and a solution was devised, products selected and rolled-out. Today, IT is often not hearing about these solutions until they’ve been piloted by shadow IT. Operating in this environment requires a shift in focus from deciding and selecting to sheparding and shaping. While IT does not want to discourage the valuable service shadow IT provides, it can’t let the process run amuck. IT must keep its head in the game and make sure these situational solutions are coordinated so they can be integrated and managed effectively. I often hear folks talk about a day when it won’t matter what disparate groups bring into the organization because everything will “just talk to each other through XML”. OK, when that day gets here, I’m all for it. However, until then, IT will needs to recognize its evolving role from systems dictator to user educator, guide and occasionally, bouncer.

Is Web 2.0 A Disaster?

I received my daily CIO Insight Quick Facts email today and was sucked in by the provocative subject line; “Is Web 2.0 A Disaster?” However, when I clicked the embedded link, the “disaster” trumpeted in the headline had quickly been downgraded to; “Web 2.0: To Good To Be True?” Note to editor…these two statements are not semantically the same. Regardless, the marketing ploy worked because I was compelled to read further.

The article opens with a Basex claim stating collaborative tools are costing businesses over a half a trillion dollars in lost productivity due to worker information overload. I didn’t read the actual Basex report (because it cost $199), so I can’t say for sure this number is hogwash but…

If companies are truly experiencing these kinds of losses due to too much information, I need to shift RatchetSoft’s focus and become a knowledge management and collaboration consulting company because the market opportunity is apparently huge. Information overload is rarely the fault of the technologies that deliver the information. It’s usually the fault of one or more of the following:

1) The group asking for the information doesn’t really know what they need. Looking for a needle in a haystack assumes you know what a haystack and needle looks like.

2) The collaboration solution is not implemented properly. Opening up the spigot and letting the data flow is just the easy part of knowledge management and collaboration. Filtering it for relevancy and making available at the right time is the hard part.

3) The information itself is of poor quality. Garbage in, garbage out.

Then the article moved on to the issue of security.

“Corporate executives also worry that Web 2.0 tools pose a security threat. Fifty-two percent of 472 executives say securing and protecting sensitive data was the top barrier to adopting the tools, according to a January poll by the Economist Intelligence Unit, a research and advisory firm, for business consultancy KPMG.

Is this news top anyone? In fact, the real news here is that only 52% of these executives see security an adoption barrier. Web 2.0 is no different than any other technology that involves information. Security has been, or at least should have been, a barrier to the adoption of every technology since the invention of the network. Bottom line is security good, hacking threats, bad, very bad. We get it. Nothing specific here to Web 2.0.

Finally, despite all the doom and gloom, the article concludes with some good news for Web 2.0:

“Three of four executives said they believe collaborative tools will foster innovation in their companies. And almost 70 percent agreed that Web 2.0 will help their people work more efficiently.”

So 70% of the respondents agreed that the half a trillion dollar loss in productivity is worth the increases in innovation? Uh, OK.

I read these kinds of articles all the time (maybe I need to get out more), and often come away asking myself; “What did I just really learn?” I’m not sure because it’s hard to decipher true insight amongst the headline hyperbole, contradictory statistics and often obvious observations by pundits. Dare I say there’s too much unfiltered information coming in. Hey, maybe there’s something to this.

SOA and Enterprise Mashups - Work On Your Pitch

I read an article today on SeachSOA.com entitled; “Enterprise Mashups, SOA’s Killer App?” While I totally agree with the sentiment of the article (that business users could care less about SOA), I’m a bit surprised that this is a revelation to anyone. The bottom line is business users don’t care about technology for technology’s sake. They only care about the relevant and tangible solutions a technology delivers. And while we technologists know how SOA translates into shorter times to value and more agile systems, business users don’t want to hear pie in the sky promises. They want to see quick and powerful results.

Regardless of whether it’s ultimately enterprise mashups or some other by product of SOA, it’s incumbent upon those who introduce new technologies into an organization to be smart marketers. As smart marketers of technology, we should keep the following points in mind when trying to secure business user buy in:

1) Know your audience. All too often, technologists get caught up in the technical details that turn themselves on yet forget about the wants, desires and needs of the person whose appetite they seek to whet. The CFO and data entry manager have very different views of the world. Be mindful of these differences and tailor your pitch accordingly.

2) Sell Specific Benefits. While we technologies love to extol the virtues of agile systems and straight through processing, business people’s eyes tend to glaze over when we use such general terms. Find out what’s important to them and then discuss the technology within that context.

3) Don’t Oversell. Business users are a pretty skeptical bunch. They’ve been sold a bill of goods before and have been left holding the bag. SOA and enterprise mashups are pretty powerful concepts that don’t require hyperbole to get others excited as well. Keep your pitch relevant, specific and on point and the rest will follow. Don’t flip the Bozo bit.

Common sense, yes. Commonly performed practice by technologists…not so much.

When I first learned about SOA, it reminded of the first time I learned about ODBC (Open Database Connectivity). I thought to myself that embedding in my applications and reports the ability to hit virtually any data source regardless of vendor or platform was going to be a game changer for business users. However, I also knew that merely telling them about a software driver that enabled a programmer to access multiple databases via native programming language access calls would be greeted with cricket calls. The best way to generate enthusiasm was to show them the power of ODBC from within the report writers and Excel spreadsheets they already used. It’s one thing to tell a business user what a given technology will do for them, it’s another thing entirely when you can show them.

I remember one business manager saying to me after a demo; “…so you’re saying this ODBC thing allows me to run a report that accesses our daily sales from the field and inventory levels in our warehouse in real time using the tools I already know instead of waiting for a weekly download and import. WOW!”.

SOA and enterprise mashups in one form or another will represent one of these WOW moments for business users if we present them in terms they understand and within a relevant and important context.