Using Images


Images are very useful in vTask, as they allow you to match exactly and interact with what is seen on the screen. Knowing how to create and access images is critical to using them successfully in vTask.


Creating Images for use in vTask
There are various ways to create images for use in vTask:

• vTask Method: Capture Area, Capture Window: Use the snapshot icon to save an image of a window or a portion of the screen. The format of the image is Bitmap (BMP).
  
  
• vTask Method: Capture 16x16 RGB, Capture 32x32 RGB: Use the snapshot icon to save a square image. The format of the image is RGB in 3-digit hexadecimal ("fff"=white, "888"=gray, "000"=black, "00f"=blue, etc). This is the same format that is usually used in CSS Style Sheets. This is the quickest way to capture images of a screen region. This method also has the advantage of being stored "in-line", without any external image files.
  
  
  
  
• Other Tool Method: Use a program like Snagit to capture images
  
  
• Manual Method: Take a snapshot of the screen (PrtScrn) or window (Alt+PrtScrn) yourself, and use a program like Paint to save the image

The best image format is the Windows Bitmap (BMP). vTask can handle other image formats, such as JPEG and GIF, but they are lossy formats, meaning the image representation may not match the screen image exactly. The maximum size for an image comparison is 1600 x 1600 pixels, however it is strongly recommended to use images as small as possible.


Using Images in vTask
Once you have an image, you can use it for many vTask actions. Common uses are moving the mouse to an image, and for window identification. If you use an image of a window, the image matching will not work if the window becomes "disabled", because the on-screen representation of the window will be dimmed.

*Tip:  The image must be visible on the screen for image identification to work. If the image is covered by another window, the image will not be found.

There are many options available for locating screen images. The Properties window provides the following settings:

• Image File: The name of the physical image file (such as BMP or JPG). Numerous small icons are available in this field:
  
        Opens an existing file
        Previews the file that is selected
        Captures area, window, and screen images
        The standard System Variable lookup, which provides access to variables (instead of the actual file names)
  
  

  *Tip:  You may use a JPG image as the source for identification, however be aware that it is a lossy format meant for photographs. This means that its pixels are not exact representations of the screen pixels. Therefore, it is recommended that when using JPG images, you should: 1) Only use JPG to find other JPG images (and not screen items such as buttons); 2) Use a high color tolerance so that it matches the loss-less format of the screen. The BMP format is a loss-less (exact) image format that is preferable.

  
  You may put multiple image filenames in the "Image File" field, separated by semicolons (;) This allows you to search for images that may contain subtle changes at run-time. The image search will check each image, until it successfully locates one on the screen.
  
  For example, you can include two images for both active and inactive window configurations:
  
       
  
  
  
  
  
• Masked Image: Indicates that you want to ignore any pixels that are pink, which means that if a source image pixel is pink, any color will match it. See the Using Image Masks section below for more information and examples.
  
  
• Stop / Start Search At: Provides a way for you to set the search rectangle. The image must fall completely within the provided coordinates, or else it will not be found. A match that begins inside of the rectangle, but ends outside the boundaries, will not qualify for a match.
  
  
• Only Scan Last Window: This allows you to restrict the screen search to the region of the current window. This window will be the last window found with a previous Activate step, or the top window if no window was activated.
  
  
• Pixel Color Tolerance: Amount of Red-Green-Blue color tolerance allowed for a match. Each of the three colors has a intensity level, from 0 to 255. A gray color would be "808080" (R=128, G=128, B=128). With a Pixel Color Tolerance of "16", you can allow a slightly darker or lighter color to produce a match. A value of "16" would match anything in the range from #707070 to #909090. A value of "255" would be an unhelpful choice, since it would always match every color on the screen (#000000 - #FFFFFF). Try smaller values at first (such as 4 or 8). If it doesn't succeed with these values, then the source image is probably significantly different than what is visible on the screen.
  
  
• Required Match %: Sets the necessary percentage of pixel matches for the image identification to succeed. Set this value to 100 (or leave blank) to require an exact match. For example, if the search image is 8x8, a 90% requirement would allow 6 pixel failures and still successfully find the image.
  
  
• Slow Pixel Scan: Force the scan to check every pixel on the screen, instead of using direct video memory access. This method is much slower than the normal method, however it will work regardless of video adapter settings and bit color depth. For example, the same image may appear differently depending on the bit depth of the screen, such as 16-bit vs. 32-bit. Use this method if you encounter problems locating images. However, you should first try Pixel Color Tolerance before using this approach.
  
  

  *Tip:  When you have difficulties matching images, try switching between the different methods of Normal and Slow Pixel Scan. They each use entirely different approaches to image comparison, and one may work in situations where the other doesn't.

Using Image Masks
"Masking" an image refers to ignoring part of an image, so that it is not used for the bitmap comparison. This is useful for inner portions of an image that may change, and you do not want to be used for matching. For the current version of vTask, the mask area is represented by the color pink (#FF00FF).

For example, using the Calculator program, the following image would only match the "5" button:



However, the following masked image would match any of the number buttons:



Another example would be to mask out a portion of an entry field, so the field is found no matter what the user types in the field.

*Tip:  vTask utilizes many internal optimizations for bitmap searching, however large bitmaps, especially with masking, can be slow on large screens. To increase the efficiency of bitmap searches, be sure to use the smallest bitmap image possible to find what you are looking for.

Sample of Web Image Interaction
As mentioned in the Identifying Windows and Controls page, web-based programs do not behave as standard Windows controls and windows. To work with internet controls, image-based scripts work the best. The key to working with web images is being able to identify the smallest portion of the web page that can be used for identification.

Many times, you will need to mask out portions of the image that might change. In the example below, the email login page can be easily identified by the small second image, however it may be necessary to mask out the User Name field. This will allow the image identification to work even if another user's name appears in the edit control.



Recommended identifying image:





Matching ClearType Text

*Tip:  Be aware that some image manipulation programs will perform functions with "anti-aliasing". This is a pixel-softening technique which changes the actual color components of pixels on the edge of selections and drawings. Paint Shop Pro and Adobe Photoshop are examples of programs that do this. With an anti-aliased edge, a perfect pixel-by-pixel comparison can be done with vTask, but the "Pixel Tolerance" must be adjusted. Be sure to disable anti-aliasing when creating images to use in vTask.

Windows XP and greater includes a screen display option called "ClearType" that smooths the edges of fonts.



This feature causes individual pixels to change their color to provide a smoother image for font curves. This results in text that is easier to read, however it causes problems with image matching.

       

The best option for obtaining consistent image matching is to have a source image that matches the screen resolution (use ClearType when creating the source image). If this is not possible, the following approach may help find images when the ClearType setting is unknown:

• 1): Set the Pixel Color Tolerance to 32
• 2): Set the Required Match % to the following formula:   (see Tip below for an easier approach)
  

100 - ( ( [total image pixel count] / [number of characters * 20] ) * 100 )

This formula tries to calculate the number of extra pixels that ClearType adds or changes, based on the number of characters in your search text. This basic strategy ignores font size, which is can significantly change the values. By viewing the "5" button sample above, you can see that approximately 20 pixels have changed. In a character like "L", almost no pixels will change, however "O" may have many pixels changed. The above formula is an informal calculation based on image match testing.

*Tip:  The best approach is to use a source image that matches your screen settings. If that's not possible, an easier formula for matching ClearType text is to start with Tolerance=32 and Required %=95, and then gradually increase the Tolerance and lower the Required % value until you achieve a consistent match.

Matching Full Window Images
A common method of finding window images is to take a snapshot of the whole window, either by using the vTask "snapshot" button, or by pressing "Alt + PrintScreen" while the window is active. However, with Windows XP and greater, the upper corners of the window are transparent because they are rounded. These few pixels show that show what is behind the window are included in the image, and will change depending on whatever is behind the window at the moment.

To resolve this issue, make sure that the area behind windows stays the same color. This is not always possible, so another solution is to mask out the upper corners of the window:

    Change To:    



This will tell vTask to ignore any colors that change behind the window in these two corners. If you have problems when matching full windows, be sure to try this approach. Be sure to change the action to the correct "Masked" version of the same action.