On an interesting project we completed recently, we developed a C++/Qt desktop application which used the Windows API to control other applications. So using an XML script, a user could open or close applications or windows and control them via simulated clicks and edits. Almost everything could be controlled including editing fields, choosing items in combo boxes, reading text, etc. The application even had secure networking built in (to control applications on other computers), email capabilities when errors were detected and multithreading to detect error conditions during the processing flow.

Now this functionality isn’t completely unique since some of it is provided by commercial applications like AutoIt, though it does provide some nice functionality like networking and email that AutoIt does not. However, what truly sets it apart is that it can detect colors and it can read static drawn text. The color functionality isn’t as interesting as it basically saves to a memory bitmap and detects the color at a certain pixel, though this is not trivial to do when the window is covered by another. Many people you ask will state that reading colors of covered windows isn’t possible but this is not the case. However to read drawn text (i.e. text that does not have a window handle) is much harder. We got the inspiration for the solution from this forum post at Experts Exchange. The solution is to read through the program’s memory (yes, contrary to popular belief, Windows does allow this) and detect labels. Here we were interested in VB6 labels in particular. So we ported the solution we found from VB to C++ since we developed the application in C++ and Qt. We then found out that it only consistently found labels in the main form of the application but not secondary forms since sometimes pointers pointed to other heap allocations. So we had to keep all of the applications’ heaps around in case this occurred so we could dereference the pointers to anywhere in the applications’ memory. Once that was done we could consistently find the text. But now we wanted to know the parent window handle and we wanted the rectangle of the drawn text. So we had to scan through the memory around the label text to find something that looked like the parent handle and something that looked like top, bottom, left and right pixel values. This took a while but we were successful in doing so.

So by reverse engineering the memory of other applications we were able to pick out this really useful information. And we think this capability sets this tool apart from all the rest. The tool is not commercially available since we developed it as a specific solution for another project. However, if there is any interest, we could easily generalize this as a commercial scripting capability. If you have any need for this, please let us know!