1. The Complete Guide to Sikuli Script
  2. 1. Basics
  3. 2. Navigator to all Classes, Methods and Functions
    1. 2.1. You may be interested in these sections
    2. 2.2. Sikuli Class Index
    3. 2.3. Sikuli Methods and Functions Index
    4. 2.4. Constants related to Mouse and Keyboard Actions
    5. 2.5. Other Constants
  4. 3. How to use this document
    1. 3.1. Abbreviations and other Conventions
    2. 3.2. Reading the Method and Function definitions
    3. 3.3. Whats new in Version 0.10
    4. 3.4. Other places to get Information
      1. About Sikuli
      2. About Python and Jython
  5. 4. Global Functions
    1. 4.1. Handling Applications
      1. openApp( application )
      2. switchApp( application )
      3. closeApp( application )
      4. run( command )
      5. exit ()
    2. 4.2. Interacting with the User
      1. popup( text )
      2. input( [text] )
    3. 4.3. General Information and Settings
      1. setBundlePath( path-to-a-folder )
      2. getBundlePath()
      3. setShowActions( False | True )
      4. getOS(), getOSVersion()
      5. getClipboard()
  6. 5. Class Region
    1. 5.1. Methods of Region
    2. 5.2. Creating a Region, Setting and Getting Attributes
      1. Region( x, y, w, h ), Region( region ), Region( Rectangle )
      2. selectRegion( [text] )
      3. setX( number ), setY( number ), setW( number ), setH( …
      4. getX(), getY(), getW(), getH()
      5. setROI( x, y, w, h | rectangle ), setRect( x, y, w, h | …
      6. getROI(), getRect()
      7. getCenter()
      8. getScreen()
      9. getLastMatch(), getLastMatches()
      10. setAutoWaitTimeout( seconds ) / getAutoWaitTimeout()
    3. 5.3. Extending a Region
      1. inside()
      2. nearby( [range] )
      3. above( [range] )
      4. below( [range] )
      5. left( [range] )
      6. right( [range] )
    4. 5.4. Finding inside a Region and Waiting for a Visual Event
      1. find( PS )
      2. findAll( PS )
      3. wait( [PS], [seconds] )
      4. waitVanish( PS, [seconds] )
      5. exists( PS, [seconds] )
    5. 5.5. Observing Visual Events in a Region
      1. onAppear( PS, handler )
      2. onVanish( PS, handler )
      3. onChange( handler )
      4. observe( [seconds], [ background= False | True ] )
      5. stopObserver()
    6. 5.6. Acting on a Region
      1. click( PSMRL, [modifiers] )
      2. doubleClick( PSMRL, [modifiers] )
      3. rightClick( PSMRL, [modifiers] )
      4. hover( PSMRL )
      5. dragDrop( PSMRL, PSMRL, [modifiers] )
      6. drag( PSMRL )
      7. dropAt( PSMRL, [delay] )
      8. type( [PSMRL], text, [modifiers] )
      9. paste( [PSMRL], text )
    7. 5.7. Low Level Mouse and Keyboard Actions
      1. mouseDown( button )
      2. mouseUp( [button] )
      3. mouseMove( PSRML )
      4. getMouseLocation()
      5. keyDown( key | list of keys )
      6. keyUp( [ key | list of keys ] )
    8. 5.8. Exception FindFailed
      1. setThrowException( False | True ) / getThrowException()
    9. 5.9. Grouping Method Calls (with Region:)
  7. 6. Class Screen
    1. 6.1. Methods of Screen
    2. 6.2. Screen: Setting, Getting Attributes and Information
      1. Screen(), Screen( id )
      2. getNumberScreens()
      3. getBounds()
    3. 6.3. Screen as (Default) Region
    4. 6.4. Capturing
      1. capture( [region | rectangle | text] ), capture( x, y, w, h )
      2. selectRegion()
    5. 6.5. Multi Monitor Environments
  8. 7. Class Location
    1. 7.1. Methods of Location
    2. 7.2. Creating a Location, Setting and Getting Attributes
      1. Location( x, y )
      2. getX(), getY(), setX( number ), setY( number )
      3. offset( dx, dy )
      4. above( dy )
      5. below( dy )
      6. left( dx )
      7. right( dx )
  9. 8. Class Match
    1. 8.1. Methods of Match
    2. 8.2. Creating a Match, Getting Attributes
      1. getScore()
      2. getTarget()
    3. 8.3. Iterating over Matches after findAll()
  10. 9. Class Finder
    1. 9.1. Methods of Finder
    2. 9.2. Creating a Finder
      1. Finder( path-to-imagefile )
    3. 9.3. Using a Finder
      1. find( path-to-imagefile, [ similarity ] )
      2. hasNext()
      3. next()
  11. 10. Class Pattern
    1. 10.1. Methods of Pattern
    2. 10.2. Creating a Pattern, Setting and Getting Attributes
      1. Pattern( string )
      2. similar( similarity )
      3. exact()
      4. targetOffset( dx, dy)
      5. getFilename()
      6. getTargetOffset()
  12. 11. Class VDict
    1. 11.1. Methods/Operators/Constants of VDict
    2. 11.2. Setting it up and getting Information
      1. VDict( [ vdict | dict ] )
      2. len( vdict )
      3. keys()
    3. 11.3. Managing Items
      1. vdict[ path-to-an-imagefile ] = value
      2. del vdict[ path-to-an-imagefile ]
    4. 11.4. Searching for specific Items
      1. path-to-an-imagefile [not] in vdict
      2. vdict[ path-to-an-imagefile ]
      3. get_exact( path-to-an-imagefile )
      4. get1( path-to-an-imagefile', similarity)
      5. get( path-to-an-imagefile', similarity, number )
    5. 11.5. Constants of VDict
  13. 12. Key Constants
    1. 12.1. Key Modifiers
    2. 12.2. Special Keys
  14. 13. Class Env

The Complete Guide to Sikuli Script

Sikuli Version: 0.10
Authors:  Raimund Hocke,  Tsung-Hsiang (Sean) Chang

1. Basics

This document was set up and is being maintained by  RaiMan (Raimund Hocke) with the great support by Tsung-Hsiang (Sean) Chang (one of the Sikuli developers). If you have any questions or ideas about this document, you are welcome to directly contact RaiMan using the mail address on his  personal Sikuli Launchpad page. For questions regarding the functions and features of Sikuli itself please use the  Sikuli Questions and Answers Board. For hints and links of how to get more information and help, please see Other places to get Information in this document.

2. Navigator to all Classes, Methods and Functions

RECOMMENDATION: To make this document more accessible, we put this navigator in the first place. But before clicking around at the first time, please read through How to use this document. It helps you to understand the basic and how to find answers in this document.

2.1. You may be interested in these sections

2.2. Sikuli Class Index

2.3. Sikuli Methods and Functions Index

Note: A method is noted with its (Class), if it exists with the same name in different classes. Each section of a class also has an alphabetical list of its methods.

  • [V]
  • [Y]
  • [Z]

2.4. Constants related to Mouse and Keyboard Actions

  • Mouse Buttons (Constants of class Button)

Button.LEFT - Button.MIDDLE - Button.RIGHT

  • The following predefined constants are listed here for your convenience ( details ).
    Their usage can be found in Acting on a Region and Low Level Mouse and Keyboard Actions. The constant's names should be self-explanatory.

    • Key Modifiers
      KEY_ALT - KEY_CMD - KEY_CTRL - KEY_META - KEY_SHIFT - KEY_WIN
      A combination of these keys has to be set up using "+" or "|", e.g. KEY_SHIFT + KEY_ALT or KEY_SHIFT | KEY_ALT | KEY_CMD (used with action methods)

    • Special Keys
      to be used with type() or keyDown()/keyUp():
      ALT - BACKSPACE - DOWN - END - ENTER - ESC - DELETE - F1 - F2 - F3 - F4 - F5 - F6 - F7 - F8 - F9 - F10 - F11 - F12 - F13 - F14 - F15 - HOME - INSERT- LEFT - PAGE_DOWN - PAGE_UP - RIGHT - TAB - UP
      the following key modifier keys can only be used with keyDown()/keyUp(): CMD - CTRL - META - SHIFT - TAB - WIN
      To use these constants, write e.g. Key.ENTER. String concatenation with "+" is possible, e.g. "some text" + Key.ENTER

2.5. Other Constants

FOREVER: can be used with exists - observe - wait - waitVanish
SCREEN: a constant reference to a screen object created by Screen(0), the default/primary monitor used as the default region.

[ top of Document ]

3. How to use this document

Since Sikuli Script is built as a Jython (Python for the Java platform) library, you can use any syntax of the Python language. If you are new to programming, you can still enjoy using Sikuli to automate simple repetitive tasks without learning Python. But if you would like to write more powerful and complicated scripts, you may want to dive into the  Python language (some additional Links). The preface of a chapter briefly describes a class or a group of methods. It contains general usage and hints that apply to all methods in that chapter. We recommend you to read it before using those methods.

If you are totally new with Sikuli, it would be a good idea to just read through this document sequentially. An alternative may be to jump to the chapters that you are interested in by scanning the table of contents. In any case, it's strongly recommended to carefully read through this entry chapter Basics. A way in the middle would be, going to Class Region, then to Class Match and ending at Class Screen.

For the users of a version 0.9.x, Whats new in Version 0.10 is a good start. After that, you can go to any places of interest using the table of contents or use the navigator to browse all classes, methods and functions in alphabetical order (as every class shows its methods this way at the beginning of its chapter).

3.1. Abbreviations and other Conventions

IDE: when used, we are talking about the Sikuli IDE. ( more Information )

PSMRL: means that either a Pattern, a string as a path to an image file, a Match, a Region or a Location can be passed as a parameter.

PS: means that either a Pattern, or a string as a path to an image file can be passed as a parameter.

bundle-path: The IDE stores the images created with the IDE's capture tool together with the script in a bundle (technically, a folder myScript.sikuli or a zipped file in case of executable myScript.skl -  more Information), at the time the script is saved or exported as executable. The full path to this bundle when the script is running is called "bundle-path".

path to an image file: A string that contains the file name of an existing image file. If the file cannot be found at runtime, a FileNotFound exception will be raised. If a file name is given as a relative path to the file, it is taken as relative to the bundle-path. An absolute path will be taken as such. When working with a Sikuli script in the IDE, a string literal to an image file is shown as the image's thumbnail. If the picture can not be found, no thumbnail will be shown. Same goes for the situations where a variable contains the path to an image file, because the value of that variable is evaluated only at runtime.

Multi Monitor Environments: All descriptions in this document are based on the standard situation for only one Monitor (upper left corner of the screen is pixel position (0, 0)). However, Sikuli supports the configurations of multiple monitors. Please see the chapter Multi Monitor Environments for more details.

x, y, w, h: denote the attributes of a rectangle on the screen where x and y are the pixel coordinates of its upper left corner (position), and w and h are its width and its height (dimension).

Rectangle: When it is used as a parameter or a return value, Rectangle is an object of java's class Rectangle, which has four attributes x, y, width and height. The attributes can be accessed (readable and writable) by saying rect.x, rect.y, rect.width, and rect.height where rect is an instance of Rectangle.

x, y: denote a pixel position (x,y) on the screen, where (0,0) is the top-left corner.

True, False: are the two boolean constants as defined in the Python language.

None: means nothing as defined in the python language.

Index Base: As in most programming languages and libraries, the first element of a list or an array has the index 0. This applies to all the cases where more than one elements are accessible by using an index. For example, the primary screen in a multi-screen environment is accessible with Screen(0).

Classes: As a common convention, the name of classes are started with an uppercase letter. Therefore, saying "Region" means the class Region, whereas a region may be an existing instance/object of class Region or something else in its context. The words instance and object are used synonymously.

CONSTANTS: As a common convention, variables written completely in capital letters are constant values. Since Python does not hinder you to redefine any of the existing constants without giving any warning, we use this convention to at least get your attention that something written in capital letters should not be redefined by you. For instance, don't say Key.ENTER = "something I think it should be". On the other hand, you are actually free to do what you want, as long as you know what you are doing ;-)

Objects and its reference To keep things simple, we normally do not differentiate between the object as a concrete structure and a reference to it. If you don't know what we are talking about, don't worry. In most situations, you don't need to know what a reference really is. However, when you start to play around with copying variables, you should be aware that Python doesn't actually "copy" things. Here is an example showing what happens, when things get mixed up: 

m = findAll(something) # returns a reference to an iterator object
mSaved = m # you may think you "save" the iterator object itself to mSaved ;-)
for gotIt in m: # going through the iterator
        print gotIt
mSaved.hasNext() # will return False, because mSaved and m are actually pointed to the same object.

After the first loop, the iterator is empty (this is how it's defined). Therefore, since mSaved is just another reference to the same object, it is also empty. A possible solution for this dilemma (saving the content of an iterator of matches for later use) can be found here.

[ top of Document ]

3.2. Reading the Method and Function definitions

Each Method or Function is described using the following structure and notation:

method( parameter, [parameter], parameter1 | parameter2, *parameter )

parameter a description of parameter (for each parameter).

A description of what this method/function does. We use Windows, Mac and Linux to give hints on specialities.

returns: a description of what is returned.

sideeffects: were applicable, we talk about. 

# example

Sometimes we give examples with screenshots, which normally can be copied and pasted into Sikuli IDE.

parameter parameters in method or function calls are denoted in bold italic.

[parameter] means that this parameter can be omitted.

parameter1 | parameter2 means that either parameter1 or parameter2 can be used. If applicable, a default value for the cases, that the parameter can be omitted or the method/function has not been used before, is given as the first choice.

*parameter means that 0 or as many parameters as wanted/needed can be used.

returns: in this block we describe what the function or the method may returns. If this block is missing, nothing is returned.

[ top of Document ]

3.3. Whats new in Version 0.10

In this section, we use 0.9 to denote the old things that are no longer existing in the version 0.10.

  • New classes Region, Screen, Location and class Match redefined

As a consolidation to form a consistent interface for all methods concerning finding a visual object and acting on it, all related methods are re-defined in the new class Region. The new class Screen and the class Match inherit all methods from class Region, so their instances can be used as a region instance. The new class Location let you act on any single point on the screen, if you really want to control each pixel. (0.9: Subregion is renamed to Region and does no longer exist.)

One or more observers to monitor visual events can be set up and run either in foreground or background. In the foreground mode, your script enters an observation loop and Sikuli invokes an event handler written by you if one of these events happen. In the background mode, your script and the observers run at the same time. It can be seen as a number of simultaneous wait()'s, handled by Sikuli for you.

  • find() always returns one match (the best match).

In 0.9, find() returns multiple matches. Since 0.10, the pattern to search for, does not define the maximum number of returned matches anymore. If successful, it always returns only one match. For multiple matches, you should use findAll() instead.

If successful a reference to this iterator object is saved in the attribute lastMatches of the region, that was used to search, for later use. This attribute is accessible through getLastMatches(). (0.9 find.regions and Matches no longer available)

  • find() saves the match in lastMatch

0.9: find.region is no longer available. You can use getLastMatch() to get the last match of find().

0.9: the xxxAll commands are no longer available. Actions on multiple targets can be done with a for loop over the iterator returned by a findAll().

Sikuli will try to get the target's location from the above types automatically. For example, if a match (returned by a find) is given, Sikuli will call match.getTarget() to get the target's location, which is the center of the match by default. If a pattern is given to click, Sikuli will call find(pattern) to get the location of the best match and then convert the match to a location using the same way as a match is given. On the other hand, if a location is given, Sikuli doesn't need to convert it anymore and will just pass the location to the action. Note that the evaluated click point should be mapped on a physical screen, read Multi Monitor Environments to learn how the screen coordinate of multi-screens is handled.

  • All find() and actions can be restricted with in a region by saying region.find() or region.click().

Restricting the search within a region speeds up your script. It's very useful when you already know where to search your targets, e.g. within a web browser. If the region is omitted, the default/primary screen is used as region. Therefore, all global finds and actions actually equal to SCREEN.find() and SCREEN.click().

  • By saying "with region:", all the following finds and actions in the with block are restricted within the region.

The "with" syntax provided since Jython 2.5 gives us a cool syntax sugar to create a context for restricting the boundaries of searching. Using a with-block, you can turn a sequence of region.click() and region.find() into a simpler block: with region: click(); find(). See Grouping Method Calls (with Region:) for more details.

  • A new class Finder allows users to search visual objects in the image files you provide.

  • Screen shots are processed directly in memory. This speeds up everything and avoids temporary files.

[ top of Document ]

3.4. Other places to get Information

Links to places outside of the Sikuli project in the following are given without taking any responsibility for their contents.

About Sikuli

About Python and Jython

[ top of Document ]

4. Global Functions

closeApp - exit - getBundlePath - input - openApp - popup - run - setBundlePath - setShowActions - switchApp

This chapter describes global functions, which are not bound to any classes.

Table of Contents

[ Navigator ] - [ top of Document ]

4.1. Handling Applications

closeApp - exit - openApp - run - switchApp

openApp( application )

application: The name of an application (case-insensitive) that can be found in the environment variable PATH, or be the full path to an application (Windows: use double backslash \\ for the path separator.)

Opens the application application and brings it to the frontmost. 

openApp("cmd.exe") # Windows: found through PATH
openApp("c:\\Program Files\\Mozilla Firefox\\firefox.exe") # windows: full path specified
openApp("Safari") # Mac: opens Safari

[ Handling Applications ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

switchApp( application )

application: The name of an application (case-insensitive).

Switches to application application and brings it to the frontmost. If the application is not running, it will be launched by openApp().

Mac: if the app has more than one windows opened, Sikuli will bring all of them to the frontmost without changing their z-order. Windows: application is searched in the text on the title bar of the running applications. So if you are sure that your app is running, application need not be an application's name. Moreover, if the windows of one app have different titles, you can select one of those windows by search their title. (not possible on Mac) 

switchApp("cmd.exe") # Windows: switches to open command prompt or starts one
switchApp("c:\\Program Files\\Mozilla Firefox\\firefox.exe") # windows: opens a new browser window !! (since text cannot be found in the window title)
switchApp("mozilla firefox") # windows: switches to the frontmost open browser window (no window open: does nothing !!)
switchApp("Safari") # Mac: switches to Safari or starts it

[ Handling Applications ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

closeApp( application )

application: The name of an application (case-insensitive).

Closes the given application application. It does nothing if no opened window (Windows) or running app (Mac) can be found.

Note: On Windows: see note with switchApp(). The whole application owning the matching window will be closed. 

closeApp("cmd.exe") # Windows: closes an open command prompt
closeApp("c:\\Program Files\\Mozilla Firefox\\firefox.exe") # windows: does nothing, since text cannot be found in the window title
closeApp("mozilla firefox") # windows: stops firefox including all its windows
closeApp("Safari") # Mac: closes Safari including all its windows

[ Handling Applications ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

run( command )

command: a command, that can be run from the command line.

Executes the command command. The script waits for completion.

[ Handling Applications ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

exit ()

Stops the script gracefully at this point.

[ Handling Applications ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

4.2. Interacting with the User

input - popup

text: a string that is used as a message

Displays a dialog box with an Ok button and text as message. The script waits for the user to click Ok. 

popup("Hello World!\nHave fun with Sikuli!") # \n can break a line.

[ Interacting with the User ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

input( [text] )

text: a string that is used as a message. If omitted, it is left blank.

Displays a dialog box with an input field, a Cancel button, an OK button and text as message. The script waits for the user to click either Cancel or Ok.

returns:

  • None if the user clicks Cancel.
  • the text that the user entered into the input field, if Ok is clicked. An empty string is returned, if nothing was entered. 
name = input("Please enter your name to log in:") # we save the returned text into a variable "name" for later use (e.g. paste())

[ Interacting with the User ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

4.3. General Information and Settings

getClipboard - getOS - getOSVersion - getBundlePath - setBundlePath - setShowActions

setBundlePath( path-to-a-folder )

path-to-a-folder a fully qualified path to a folder containing your images used for finding patterns. Windows: use double backslashes.

Sets the path for searching images in all Sikuli Script methods. Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli). Therefore, you should use this function only if you really know what you are doing. Using it generally means that you would like to take care of your captured images by yourself.

[ General Settings ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

getBundlePath()

returns: a string containing a fully qualified path to a folder containing your images used for finding patterns. Note: Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli). You may use this function if, for example, to package your private files together with the script or to access the picture files in the .sikuli bundles for other purposes. Sikuli only gives you to access to the path name, so you may need other python modules for I/O or other purposes.

[ General Settings ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

setShowActions( False | True )

If set to True, when a script is run, Sikuli shows a visual effect on the spot where the action will take place before executing actions (e.g. click, dragDrop, type, etc) for about 2 seconds . The default setting is False.

[ General Settings ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

getOS(), getOSVersion()

  • Env.getOS() returns the type of system your script is running (a static method of Class Env).

returns: one of the following constants: OS.MAC, OS.WINDOWS, OS.LINUX.

  • Env.getOSVersion() returns the version number of the system the script is running on. It does not tell you the type of system as such (use getOS())

returns: a string

The example shows how to check where you are. 

# on a Mac
myOS = Env.getOS() # you have a reference to the constant OS.MAC
myVer = Env.getOSVersion() # the specific version number

if myOS == OS.MAC:
   print "Mac " + myVer # prints e.g.: Mac 10.6.3
else:
   print "Sorry, not a Mac"

myOS = str(Env.getOS()) # you have a reference to a string containing "MAC"
if myOS == "MAC" or myOS.startswith("M"): # to show the possibilities
   print "Mac " + myVer # prints e.g.: Mac 10.6.3
else:
   print "Sorry, not a Mac"

[ General Settings ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

getClipboard()

Usage: Env.getClipboard() (a static method of Class Env)

returns: the content of the Clipboard if it is text, otherwise an empty string.

Note: Be careful, when using getClipboard() together with paste(): since paste internally uses the clipboard to transfer text to other applications, the clipboard will contain what you just pasted. Therefore, if you need the content of the clipboard, you should call getClipboard() before using paste().

Tip: When the clipboard content was copied from a web page that mixes images and text, you should be aware, that there may be whitespace characters around and inside your text, that you did not expect. In this case, you can use Env.getClipboard().strip() to get rid of surrounding whitespaces.

[ General Settings ] - [ Global Functions ] - [ Navigator ] - [ top of Document ]

5. Class Region

Table of Contents

[ Navigator ] - [ top of Document ]

A region is a rectangular section on a screen, which is defined by

  1. a location (x, y) of its upper left corner as a distance relative to the upper left corner of the screen (0, 0),
  2. and its dimension (w, h) as its width and height.

x, y, w, h are integer numbers counting a distance in pixels.

A region knows nothing about its visual content (windows, pictures, graphics, text, ...). It only knows the position on the screen and its dimension.

New regions can be created, based on an existing region: you can extend a region in all directions or get the adjacent rectangle up to the bounds of the screen horizontally or vertically.

The visual content of a region is evaluated by using methods like find(), which looks for a given rectangular pixel pattern within the region. The matching content in the region has a similarity between 0 (not found) and 1 (found and it is per pixel exactly matches to the pattern). The find can be advised, to search with a minimum similarity, so that some minor variations in shape and color can be ignored. If nothing else is specified, Sikuli searches with a minimum similarity of 0.7, which does what is expected in general cases.

Find operations return a match, which has all the attributes and methods as a region has and can be used in exactly the same way (e.g. to find or click another target within it). A match has the dimension of the pattern used for searching and also knows the position where it was found and its similarity score. A region preserves the best match of the last successful find operation and all matches of the last successful findAll() (accessible with getLastMatch()/getLastMatches()). You can wait for patterns to show up using wait(), to vanish using waitVanish(), or just check whether a pattern exists without handling exceptions.

Sikuli 0.10 begins to support visual event driven programming. You can tell a region to observe that something appears, vanishes, or changes. It's possible to wait for the completion of an observation or let it run in the background while your script is continuing. When one of the visual events happens, a handler in your script is called. Each region has one observer and each observer can handle multiple visual events. It's your responsibility to stop an observation.

You can act on a region by simulating mouse and keyboard actions. You have the choice between acting on spots that are evaluated from previous finding operations (e.g. a match), or act on a pattern that will be found implicitly. To simulate more complicated actions with special applications like games or graphics, low level actions for mouse and keyboard are available.

To support even more sophisticated and robust scripts, you may want to handle the FindFailed exception, which is thrown when a find operation is failed.

Operations that target on the same region can be grouped together using Python's "with" statement.

Screen is another class that inherits all attributes and methods of class Region. Through class Screen, you can access to the screen dimensions and different monitors, if you have more than one. It also gives you a default region to act on without specify one. Normally you would have to say region.find(image), where region is usually the whole screen. So as a convenience, saying find(image) would simply act on the whole screen (actually the default/primary screen), without specifying the default SCREEN every time. On the other hand, this may slow down searching speed, because finding a target on the whole screen is time-consuming. Therefore, to speed up processing, saying region.find() can restrict the search to a specified smaller rectangle that is usually the application's main window you are interested. Another possibility is to use setROI() to restrict the search for all following find operations to a smaller region than the whole screen.

PSMRL: means that either a Pattern, a string as a path to an image file, a Match, a Region or a Location can be passed as a parameter.

PS: means that either a Pattern, or a string as a path to an image file can be passed as a parameter.

applicable for Screen or Match or Screen and Match Though all methods of class Region can be used with objects of the classes Screen and Match, not every method makes really sense with these classes ( e.g. Sccreen(0).below(), find(PS).setRect(rectangle) ). So if mentioned, this means that the usage of this method with a screen object and/or a match object makes sense in applicable situations.

Note on Multi Monitor Environments: In situations where more than one monitor are activated and you want to act with regions that are not located on the default screen (primary monitor in this case), special aspects have to be taken into account. Therefore, before starting to work with Sikuli on more than one monitor, please read the chapter Multi Monitor Environments at first.

[ Navigator ] - [ top of Document ]

5.1. Methods of Region

above - below - click - drag - dragDrop - dropAt - doubleClick - exists - find - findAll - getCenter - getH - getLastMatch - getLastMatches - getRect - getROI - getScreen - getW - getX - getY - hover - inside - keyDown - keyUp - left - mouseDown - mouseMove - mouseUp - nearby - observe - onAppear - onVanish - onChange - paste - right - rightClick - selectRegion - setAutoWaitTimeout - setH - setRect - setROI - setThrowException - setW - setX - setY - stopObserver - type - wait - waitVanish

Note: In case of having more than one Monitor active, read Multi Monitor Environments before.

[ Navigator ] - [ top of Document ]

5.2. Creating a Region, Setting and Getting Attributes

getCenter - getH - getLastMatch - getLastMatches - getRect - getROI - getScreen - getW - getX - getY - selectRegion - setAutoWaitTimeout - setH - setRect - setROI - setW - setX - setY

In this chapter, you can find information on how to create a new region object. Some of the attributes of a region object can be accessed directly or via a method call. Here you will find the HowTo's.

Region( x, y, w, h ), Region( region ), Region( Rectangle )

x, y, w, h: the attributes of a rectangle.

region: an existing region object.

rectangle: an existing object of java class Rectangle.

In addition to creating a region by using the tool provided by the IDE, a region can be created by specifying a rectangle. This is how the visual representation in the IDE of such a region is internally set up in the script. A region can also be created by users in run-time using selectRegion().

You can create a region by given another region. This just duplicates the region into a different and new object. This can be useful, if you need the same region with different attributes, such as observation loop or whether throwing an exception when finding fails. Another way to create a region is to specify a rectangle object or to extend an existing region.

returns: a new region object 

myReg = Region(0, 0, 500, 500) # a new region in upper left part of screen, with width=500 and height=500
myReg1 = Region( myReg ) # new region with the same rectangle as myReg

# using a rectangle
rect = myReg1.getRect() # returns a rectangle object
rect.x += 100 # adds 100 to its x coordinate
myReg2 = Region( rect )

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

selectRegion( [text] )

In fact, selectRegion() is a method of Class Screen, but since it creates a region, it's mentioned here too.

text: is displayed for about 2 seconds in the middle of the screen. If text is omitted, the default "Select a region on the screen" is displayed.

The interactive capture mode is entered and allows the user to select a region the same way as using the selection tool in the IDE. You may have to check the result, since the user may cancel the capturing.

returns: a new region object or None, if the user cancels the capturing process.

applicable for Screen

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

setX( number ), setY( number ), setW( number ), setH( number )

number: an integer value

The respective attribute of the region is set to the new value. This effectively moves the region around and/or changes its dimension.

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

getX(), getY(), getW(), getH()

returns: the respective attribute of the region as an integer value.

applicable for Screen and Match

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

setROI( x, y, w, h | rectangle ), setRect( x, y, w, h | rectangle )

x, y, w, h: the attributes of a rectangle

rectangle: a rectangle object

Both methods are doing exactly the same: setting position and dimension to new values. The motivation for two names is to make scripts more readable: setROI() is intended to shrink a screen object to speed up processing searches (region of interest), whereas setRect() should be used to redefine a region (which could be enlarging it).

returns: None

applicable for Screen

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

getROI(), getRect()

Both are doing the same: for motivation to have two, see above.

returns: the rectangle attributes of the region as a rectangle object

applicable for Screen and Match 

myRect = myReg.getRect() # myRect is an object of java class Rectangle
(x, y, w, h) = (myRect.x, myRect.y, myRect.width, myRect.height) # the 4 attributes are assigned to different variables
print getROI() # shows "java.awt.Rectangle[x=0,y=0,width=1280,height=800]" (current region of interest of primary screen)

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

getCenter()

Returns the pixel position of the center of that region: x + w/2, y + h/2. Fractions are rounded down to integer.

returns: an object of class Location.

applicable for Screen and Match 

loc = Region(0, 0, 101, 201).getCenter() # loc will contain position (50, 100)

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

getScreen()

Returns the screen object that contains this region. It only makes sense in Multi Monitor Environments, since it always return the default screen in a single monitor environment.

returns: a new screen object

applicable for Match

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

getLastMatch(), getLastMatches()

All successful find operations ( explicit like find() or implicit like click() ), store the best match into lastMatch of the region that was searched. findAll() store all found matches into lastMatches of the region that was searched as an iterator.

To access these attributes use region.getLastMatch() or region.getLastMatches() respectively.

How to use the iterator object returned by getLastMatches() is documented here.

returns: the best match as a match object respectively one or more match objects as an iterator object.

applicable for Screen and Match

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

setAutoWaitTimeout( seconds ) / getAutoWaitTimeout()

  • setAutoWaitTimeout( seconds )

seconds: a number, which can have a fraction. The internal granularity is milliseconds.

The default is 3.0 seconds.

Sets the maximum waiting time in seconds for all following find operations in the respective region (by default SCREEN is used, if region is not specified). It enables all find operations to wait for the given pattern to appear until the specified amount of time has elapsed. As such it lets find() work like wait(), without being able to set an individual timeout value for a specific find operation.

  • getAutoWaitTimeout()

returns: the current value of the maximum waiting time for all following find operations in the respective region (by default SCREEN is used, if region is not specified).

[ Creating a Region, Setting and Getting Attributes ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

5.3. Extending a Region

above - below - inside - left - nearby - right

These methods (exception inside()) return a new region object, that is constructed based on the specified region (sometimes called spatial operators). The range parameter, if given as positive integer number, restricts the dimension of the new region (width and/or height respectively) based on that value. If range is not specified, the new region extends to the respective boundary of the screen the given region belongs to. An exception is nearby(), which uses 50 as its default range.

Note: In all cases the new region does not extend beyond any boundary of the screen that contains the given region.

inside()

inside() can be used to make scripts more readable. region.inside().find() is totally equivalent to region.find().

returns: the region itself

applicable for Match

[ Extending a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

nearby( [range] )

range: positive integer number

The new region is extended in all directions by range pixels with the same center as the given region. If not given, range is set to 50 pixels by default.

returns: a new region (have a look)

applicable for Match

[ Extending a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

above( [range] )

The new region is constructed above the given one starting from its top border extending range pixels towards the top boundary of the screen containing the given region. If range is omitted, it extends to the boundary of the screen. It has the same width as the given region and the same rectangle x-value.

returns: a new region (have a look)

applicable for Match

[ Extending a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

below( [range] )

The new region is constructed below the given one starting from its bottom border extending range pixels towards the bottom boundary of the screen containing the given region. If range is omitted, it extends to the boundary of the screen. It has the same width as the given region and the same rectangle x-value.

returns: a new region (have a look)

applicable for Match

[ Extending a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

left( [range] )

The new region is constructed left of the given one starting from its left border extending range pixels towards the left boundary of the screen containing the given region. If range is omitted, it extends to the boundary of the screen. It has the same height as the given region and the same rectangle y-value.

returns: a new region (have a look)

applicable for Match

[ Extending a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

The new region is constructed right of the given one starting from its right border extending range pixels towards the right boundary of the screen containing the given region. If range is omitted, it extends to the boundary of the screen. It has the same height as the given region and the same rectangle y-value.

returns: a new region (have a look)

applicable for Match

5.4. Finding inside a Region and Waiting for a Visual Event

exists - find - findAll - wait - waitVanish

In addition to acting on visual objects, finding them is also one of the core functions of Sikuli. As a default, if the visual object cannot be found, Sikuli will stop the script by raising an exception FindFailed. This follows the standards of the Python language so that you can handle such exceptions using  try ... except. If you are not used to programming and just want to bypass the exception handling, you can read this section about exception FindFailed.

PS: means, that either a Pattern or a string as a path to an image file can be used as parameter. A find operation is successful, if the given image is found with the given minimum similarity. Similarity is a value between 0 and 1 to specify how likely the given image looks like the target. By default, the similarity is 0.7 if an image rather than a pattern object with a specific similarity is given to find().

If a find operation is successful, the returned match is additionally stored internally with the region that was used for the search. So instead of using a variable to store the match (m = find()), you can use getLastMatch() to access it afterwards. Unsuccessful find operations will leave these values unchanged. (This only happens when using exists(), exception handling or when running with setThrowException(''False'')).

Normally all these region methods are used as reg.find(PS), where reg is a region object. If written as find(PS) it acts on the default screen, which is an implicit region in this case. But sometimes it's a good idea to use region.find() to restrict the search to a smaller region in order to speed up processing.

If you have multiple monitors, please read Multi Monitor Environments.

Note on IDE: Capturing is a tool in the IDE, to quickly set up images to search for. These images are named automatically by the IDE and stored together with the script, at the time it is saved (we call the location in the file system bundle-path). Behind the curtain, the images itself are specified simply by using a string containing the file name (path to an image file).

find( PS )

PS: a pattern object or a string (path to an image file)

Looks for a particular GUI element, which is seen as the given image. The given file name of an image specifies the element's appearance. It searches within the region and returns the best match, which shows a similarity greater than the minimum similarity given by the pattern. If no similarity was set for the pattern by Pattern.similar() before, a default minimum similarity of 0.7 is set automatically. If no match is found with the minimum similarity or greater, the find fails (raises exception FindFailed or returns None).

If autoWaitTimeout (setAutoWaitTimeout()) is set to a non-zero value, find() just acts as a wait().

returns: a match object that contains the best match. In case that exception handling for FindFailed is switched off by setThrowException(False), None is returned if nothing is found. (Note: By default, the exception handling of FindFailed is turned on).

Sideeffects:

  • FindFailed: If the find fails (no match, whose similarity is equal or greater than the minimum similarity of the pattern, can be found) and exception handling is turned on (which is the the default), an exception FindFailed is raised. If the script does not handle the exception, the script is stopped, with a message about the exception.
  • lastMatch: the best match can be accessed using getLastMatch() afterwards

applicable for Screen and Match

[ Finding inside a Region and Waiting for a Visual Event ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

findAll( PS )

PS: a pattern object or a string (path to an image file)

Repeatedly looks for the pattern, until no match can be found anymore, that meets the requirements for a single find() with the specified pattern ( look for details ).

returns: one ore more match objects as an iterator object. How to iterate through is documented here. In case that exception handling for FindFailed is switched off by setThrowException(False) None is returned, if nothing is found. (Note: By default at script start exception handling of FindFailed is turned on).

Sideeffects:

  • FindFailed: If the find fails (no match can be found, whose similarity is equal or greater than the minimum similarity of the pattern) and exception handling is turned on (which is the default) an exception FindFailed is raised. If the script does not handle the exception, the script is stopped, with a message about the exception.
  • lastMatches: a reference to the returned iterator object containing the found matches is stored with the region that was searched. It can be accessed using getLastMatches() afterwards. How to iterate through an iterator of matches is documented here.

applicable for Screen and Match

[ Finding inside a Region and Waiting for a Visual Event ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

wait( [PS], [seconds] )

PS: a pattern object or a string (path to an image file)

seconds: a number, which can have a fraction, as maximum waiting time in seconds. The internal granularity is milliseconds. If not specified, the auto wait timeout value is used. Use the constant FOREVER to wait for an infinite time.

If PS is not specified, the script just pauses for the specified amount of time. ( it's still possible to use sleep( seconds ) instead, but this is deprecated. )

PS specified: Keeps searching the given pattern in the region until the image appears ( would have been found with find(PS) ) or the specified amount of time has elapsed. At least one find operation is performed, even if 0 seconds is specified.

returns: a match object that contains the best match. If exception handling for FindFailed is switched off by setThrowException(False), None is returned, if nothing is found within the specified waiting time. (Note: By default, the exception handling of FindFailed is turned on.)

Sideeffects:

  • FindFailed: If the find fails (no match can be found, whose similarity is equal or greater than the minimum similarity of the pattern within the specified waiting time) and exception handling is turned on (which is the default), an exception FindFailed is raised. If the script does not handle the exception, the script is stopped with a message about the exception.
  • lastMatch: the best match can be accessed using getLastMatch() afterward.

applicable for Screen and Match

[ Finding inside a Region and Waiting for a Visual Event ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

waitVanish( PS, [seconds] )

PS: a pattern object or a string (path to an image file)

seconds: a number, which can have a fraction, as maximum waiting time in seconds. The internal granularity is milliseconds. If not specified, the actual auto wait timeout value is used. Use the constant FOREVER to wait for an infinite time.

Keeps searching the given pattern in the region until the image vanishes (can not be found with find(PS) any longer) or the specified amount of time has elapsed. At least one find operation is performed, even if 0 seconds is specified.

returns:

  • True if the pattern vanishes within the specified waiting time.
  • False if the pattern stays visible until the waiting time has elapsed.

applicable for Screen and Match

[ Finding inside a Region and Waiting for a Visual Event ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

exists( PS, [seconds] )

PS: a pattern object or a string (path to an image file)

seconds: a number, which can have a fraction, as maximum waiting time in seconds. The internal granularity is milliseconds. If not specified, the actual auto wait timeout value is used. Use the constant FOREVER to wait for an infinite time.

Does exactly the same as wait(), but no exception is raised in case of FindFailed. So it can be used to symplify scripting in case that you only want to know wether something is there or not to decide how to proceed in your workflow. So it is typically used with an if statement. At least one find operation is performed, even if 0 seconds is specified. So specifying 0 seconds saves some time, in case there is no need to wait, since its your intention to get the information "not found" directly.

returns: a match object that contains the best match. None is returned, if nothing is found within the specified waiting time.

Sideeffects:

  • lastMatch: the best match can be accessed using getLastMatch() afterwards

applicable for Screen and Match

[ Finding inside a Region and Waiting for a Visual Event ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

5.5. Observing Visual Events in a Region

observe - onAppear - onVanish - onChange - stopObserver

This feature allows to some extent the implementation of visual event driven programming.

You can tell a region to observe that something appears or vanishes or the content changes at all. Using the methods onAppear(), onVanish() and onChange(), you register an event observer that starts its observation when you call observe(). Each region object can have exactly one observation active and running. For each observation, you can register as many event observers as needed. So you can think of it as grouping some wait() and waitVanish() together and have them processed simultanouesly, while you are waiting for one of these events to happen.

It's possible to let the script wait for the completion of an observation or let it run in the background, while your script is continuing. With a timing parameter you can tell observe() to stop observation anyway after the given time.

When one of the visual events happens, an event handler written by you is called. An event handler is a function contained in your script and expects an event object as a parameter. During the processing in your handler, the observation is paused until your handler has ended. Information between the main script and your handlers can be given forward and backward using global variables.

It's your responsibility to stop an observation. This can either be done by calling region.stopObserver() or by starting observe() with a timing parameter.

Since you can have as many region objects as needed and each region can have one observation active and running. So theoretically it's possible to have as many visual events being observered at the same time as needed. But in reality, the number of observation is still limited by the system resources available to Sikuli at that time.

Be aware, that every observation is a number of different find operations that are processed repeatedly. So to speed up processing and keep your script acting, you may want to define a region for observation as small as possible.

PS: means, that either a Pattern or a String as a path to an image file can be used as parameter.

handler: as a parameter in the following methods, you have to specify the name of a function, which will be called by the observer, in case the observed event happens. The function itself has to be defined in your script before using the method that references the function. The existance of the function will be checked before starting the script. So to get your script running, you have to have at least the following statements in your script: 

def myHandler(event): # you can choose any valid function name
        # event: can be any variable name, it references an event object
        pass # add your statements here

onAppear("path-to-an-image-file", myHandler) # or any other onEvent()
observe(10) # observes for 10 seconds

Normally all the region methods are used as reg.onAppear(PS), where reg is a region object. If written as onAppear(PS) the different repeatedly performed implicit find operations operate on the default screen being the implicit region in this case. But using region.onEvent() will restrict the search to the region's rectangle and speed up processing, if region is significantly smaller than the whole screen.

Note: In case of having more than one Monitor active, read Multi Monitor Environments before.

Note on IDE: Capturing is a tool in the IDE, to quickly set up images to search for. These images are named automatically by the IDE and stored together with the script, at the time it is saved (we call the location in the file system bundle-path). Behind the curtain the images itself are specified by using a string containing the file name (path to an image file).

[ Navigator ] - [ top of Document ]

onAppear( PS, handler )

PS: a pattern object or a string (path to an image file)

handler: the name of a function contained in your script.

With the given region you register an observer, that should wait for the pattern to be there or to appaear and is activated with the next call of observe(). In the moment the internal find operation on the given pattern is successful during observation, your handler is called and the observation is paused until you return from your handler.

applicable for Screen and Match

[ Observing Visual Events in a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

onVanish( PS, handler )

PS: a pattern object or a string (path to an image file)

handler: the name of a function contained in your script

With the given region you register an observer, that should wait for the pattern to be not there or to vanish and is activated with the next call of observe(). In the moment the internal find operation on the given pattern fails during observation, your handler is called and the observation is paused until you return from your handler.

applicable for Screen and Match

[ Observing Visual Events in a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

onChange( handler )

handler: the name of a function contained in your script

With the given region you register an observer, that should wait for the visual content of the given region to change and is activated with the next call of observe(). In the moment the visual content changes during observation, your handler is called and the observation is paused until you return from your handler.

applicable for Screen and Match

[ Observing Visual Events in a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

observe( [seconds], [ background= False | True ] )

seconds: a number, which can have a fraction, as maximum observation time in seconds. Omit it or use the constant FOREVER to tell the observation to run for an infinite time (or until stopped by a call of stopObserve()).

background= False | True : when set to True, the observation will be run in the background and processing of your script is continued immediately. Otherwise the script is paused until the completion of the observation.

When called, the observation is started. For each region object, only one observation can be running at a given time.

applicable for Screen and Match

[ Observing Visual Events in a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

stopObserver()

When called, a running observation of the relevant region is stopped.

  • as reg.stopObserver(), reg must be a valid reference to a region object.
  • If you are not sure, wether you have access to this reference or do not know which one to use (e.g. using the same handler for different observations) you can use event.region.stopObserver, where event is the one parameter that is defined with the handler (can be any other valid variable name). Obviously only availble inside a handler.

applicable for Screen and Match

[ Observing Visual Events in a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

5.6. Acting on a Region

click - drag - dragDrop - dropAt - doubleClick - hover - paste - rightClick - type

Besides finding visual objects on the screen, acting on these elements is one of the kernel operations of Sikuli. Mouse actions can be simulated as well as pressing keys on a keyboard.

The place on the screen, that should be acted on (in the end just one specific pixel, the click point), can be given either as a pattern PS like with the find operations or by directly referencing a pixel as center of a region object (match or screen also), the target pixel connected with a match or a pixel location. Since all these choices can be used with all action methods as needed, they are abbreviated and called like this:

PSMRL: which means, that either a Pattern or a string as a path to an image file or a Match or a Region or a Location can be used as parameter, in detail:

  • P: pattern: a pattern object. An implicit find operation is processed first. If successful, the center of the resulting matches rectangle is the click point. If the pattern object has a target offset specified, this is used as click point instead.
  • S: string: a path to an image file. An implicit find operation with the default minimum similarity 0.7 is processed first. If successful, the center of the resulting match object is the click point.
  • M: match: a match object from a previous find operation. If the match has a target specified it is used as the click point, otherwise the center of the matches rectangle.
  • R: region: a region object whose center is used as click point.
  • L: location: a location object which by definition represents a point on the screen that is used as click point.

modifiers It is possible to press the so called key modifiers together with the mouse operation or when simulating keyboard typing. The respective parameter is given by one or more predefined constants. If more than one modifier is necessary, they are combined by using "+" or "|".

Be aware: when using PS as parameter, internally there has to be processed an implicit find operation before, so the aspects of find operations and of find(''PS'') apply. If the find operation was successful, the match that was acted on, can be recalled using getLastMatch(). Especially remember the fact, that as a default, Sikuli will stop the script, if the visual object cannot be found, by raising an exception FindFailed. This is done according to the standards of the Python language, which allows you to handle such exceptions. If you are not really used to programming and do not have a good knowledge of Python, it may be helpful, to first read about exception FindFailed.

Normally all these region methods are used as reg.click(PS), where reg is a region object. If written as click(PS) the implicit find is done on the default screen being the implicit region in this case. But using region.click() will restrict the search to the region's rectangle and speed up processing, if region is significantly smaller than the whole screen.

Note on IDE: Capturing is a tool in the IDE, to quickly set up images to search for. These images are named automatically by the IDE and stored together with the script, at the time it is saved (we call the location in the file system bundle-path). Behind the curtain the images itself are specified by using a string containing the file name (path to an image file).

Note: If you need to implement more sophisticated mouse and keyboard actions look at Low Level Mouse and Keyboard Actions.

Note: In case of having more than one Monitor active, read Multi Monitor Environments before.

Note on Mac: it might be necessary, to use switchApp() before, to prepare the application for accepting the action

[ Navigator ] - [ top of Document ]

click( PSMRL, [modifiers] )

PSMRL: a pattern, a string, a match, a region or a location that evaluates to a click point (read for details)

modifiers: one or more key modifiers (read for details)

Performs a mouse click on the click point using the left button.

returns: the number of performed clicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed. This would be the case, if using PS (yields an implicit find), the find fails and you have switched of exception FindFailed to be raised. Otherwise the script is stopped with a FindFailed exception (read details).

Sideeffect: when using PS, the match can be accessed using getLastMatch() afterwards

[ Acting on a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

doubleClick( PSMRL, [modifiers] )

PSMRL: a pattern, a string, a match, a region or a location that evaluates to a click point (read for details)

modifiers: one or more key modifiers (read for details)

Performs a mouse double click on the click point using the left button.

returns: the number of performed double clicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed. This would be the case, if using PS (yields an implicit find), the find fails and you have switched of exception FindFailed to be raised. Otherwise the script is stopped with a FindFailed exception (read details).

Sideeffect: when using PS, the match can be accessed using getLastMatch() afterwards

[ Acting on a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

rightClick( PSMRL, [modifiers] )

PSMRL: a pattern, a string, a match, a region or a location that evaluates to a click point (read for details)

modifiers: one or more key modifiers (read for details)

Performs a mouse click on the click point using the right button.

returns: the number of performed clicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed. This would be the case, if using PS (yields an implicit find), the find fails and you have switched of exception FindFailed to be raised. Otherwise the script is stopped with a FindFailed exception (read details).

Sideeffect: when using PS, the match can be accessed using getLastMatch() afterwards

[ Acting on a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

hover( PSMRL )

PSMRL: a pattern, a string, a match, a region or a location that evaluates to a click point (read for details)

Moves the mouse pointer to the click point and does nothing else.

returns: the number 1 if the mousepointer could be moved to the click point. If using PS (yields an implicit find), the find fails and you have switched of exception FindFailed to be raised, a 0 (integer null) is returned. Otherwise the script is stopped with a FindFailed exception (read details).

Sideeffect: when using PS, the match can be accessed using getLastMatch() afterwards

[ Acting on a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

dragDrop( PSMRL, PSMRL, [modifiers] )

PSMRL: a pattern, a string, a match, a region or a location that evaluates to a click point (read for details)

modifiers: one or more key modifiers (read for details)

A complete drag-and-drop operation is performed. The first PSMRL is the starting click point, the second PSMRL the target clickpoint. When the operation does not perform as expected (usually caused by timing problems due to delayed reactions of applications), it may be necessary to use the combination drag()/dropAt().

Sideeffect: when using PS as the second parameter or both, the target match can be accessed using getLastMatch() afterwards. If only the first parameter is given as PS, this match is returned by getLastMatch().

Note: If you need to implement more sophisticated mouse and keyboard actions look at Low Level Mouse and Keyboard Actions.

[ Acting on a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

drag( PSMRL )

PSMRL: a pattern, a string, a match, a region or a location that evaluates to a click point (read for details)

The mousepointer is moved to the click point and the left mouse button is pressed and held, until another mouse action is performed (e.g. a dropAt() afterwards). This is nomally used to start a drag-and-drop operation.

returns: the number 1 if the operation could be performed. If using PS (yields an implicit find), the find fails and you have switched of exception FindFailed to be raised, a 0 (integer null) is returned. Otherwise the script is stopped with a FindFailed exception (read details).

Sideeffect: when using PS, the match can be accessed using getLastMatch() afterwards

Note: If you need to implement more sophisticated mouse and keyboard actions look at Low Level Mouse and Keyboard Actions.

[ Acting on a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

dropAt( PSMRL, [delay] )

PSMRL: a pattern, a string, a match, a region or a location that evaluates to a click point (read for details)

delay: a number, which can have a fraction, as waiting time in seconds. The internal granularity is milliseconds.

The mousepointer is moved to the click point. After waiting for delay seconds the left mouse button is released. This is normally used to finalize a drag-and-drop operation. If it is necessary to visit one ore more click points after dragging and before dropping, you can use mouseMove() inbetween.

returns: the number 1 if the operation could be performed. If using PS (yields an implicit find), the find fails and you have switched of exception FindFailed to be raised, a 0 (integer null) is returned. Otherwise the script is stopped with a FindFailed exception (read details).

Sideeffect: when using PS, the match can be accessed using getLastMatch() afterwards

Note: If you need to implement more sophisticated mouse and keyboard actions look at Low Level Mouse and Keyboard Actions.

[ Acting on a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

type( [PSMRL], text, [modifiers] )

PSMRL: a pattern, a string, a match, a region or a location that evaluates to a click point (read for details)

text: a string containing the characters, that should be typed from left to right

modifiers: one or more key modifiers (read for details)

Simulates keyboard typing interpreting the characters of text based on the layout/keymap of the standard US keyboard (QWERTY). Special keys (ENTER, TAB, BACKSPACE, ...) can be incorporated into text by using the constants defined in class Key using the standard string concatenation (+). If PSMRL is given, a click on the clickpoint is performed before typing, to gain the focus. (Mac: it my be necessary, to use switchApp() before, to prepare the application for accepting the click)

If PSMRL is omitted, it performs the typing on the current focused visual component (normally an input field or an menu entry that can be selected by typing something). returns: the number 1 if the operation could be performed, otherwise 0 (integer null). If using PS (yields an implicit find), the find fails and you have switched of exception FindFailed to be raised, a 0 (integer null) is returned. Otherwise the script is stopped with a FindFailed exception (read details).

Sideeffects: when using PS, the match can be accessed using getLastMatch() afterwards

Note: If you need to type international characters or you are using layouts/keymaps other than US-QWERTY, you should use paste() insteda. Since type() is rather slow, also use paste() to type longer texts.

[ Acting on a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

paste( [PSMRL], text )

PSMRL: a pattern, a string, a match, a region or a location that evaluates to a click point (read for details)

text': a string that should be pasted into the target (e.g. input field)

Pastes text using the clipboard with OS-level shortcut (Ctrl-V or Cmd-V). So afterwards your clipboard contains text. paste() is a temporary solution for typing international characters or typing on keyboard layouts other than US-QWERTY.

If PSMRL is given, a click on the clickpoint is performed before typing, to gain the focus. (Mac: it my be necessary, to use switchApp() before, to prepare the application for accepting the click)

If PSMRL is omitted, it performs the paste on the current focused component (normally an input field).

returns: the number 1 if the operation could be performed, otherwise 0 (integer null). If using PS (yields an implicit find), the find fails and you have switched of exception FindFailed to be raised, a 0 (integer null) is returned. Otherwise the script is stopped with a FindFailed exception (read details).

Sideeffects: when using PS, the match can be accessed using getLastMatch() afterwards

NOTE: Special keys (ENTER, TAB, BACKSPACE, ...) cannot be used with paste(). If needed, you have to split your complete text into two or more paste()'s and use type() for typing the special keys inbetween. Characters like \n (enter/new line) and \t (tab) should work as expected with paste().

[ Acting on a Region ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

5.7. Low Level Mouse and Keyboard Actions

getMouseLocation - keyDown - keyUp mouseDown - mouseMove - mouseUp

PSMRL: means, that either a Pattern or a String (path to an image file) or a Match or a Region or a Location can be used (e.g. as parameter).

button: when used as parameter in mouse actions, it has to be one of the constants Button.LEFT, Button.MIDDLE, Button.RIGHT or a combination using the + or | operator.

Note: In case of having more than one Monitor active, read Multi Monitor Environments before.

[ Navigator ] - [ top of Document ]

mouseDown( button )

button: it has to be one of the constants Button.LEFT, Button.MIDDLE, Button.RIGHT denoting the button to be used or a combination.

The mouse button(s) button is(are) pressed and held until another mouse action is performed.

returns: the number 1 if the operation could be performed otherwise 0 (integer null).

[ Low Level Mouse and Keyboard Actions ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

mouseUp( [button] )

button: it has to be one of the constants Button.LEFT, Button.MIDDLE, Button.RIGHT denoting the button to be used or a combination.

The mouse button(s) button is(are) released. If button is omitted, all currently pressed buttons are released.

returns: the number 1 if the operation could be performed otherwise 0 (integer null).

[ Low Level Mouse and Keyboard Actions ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

mouseMove( PSRML )

PSMRL: means, that either a Pattern or a String (path to an image file) or a Match or a Region or a Location can be used as parameter (for details look here).

The mouse pointer is moved to the target pixel, that is evaluated from PSMRL.

returns: the number 1 if the operation could be performed. If using PS (yields an implicit find), the find fails and you have switched of exception FindFailed to be raised, a 0 (integer null) is returned. Otherwise the script is stopped with a FindFailed exception (read details).

Sideeffects:

  • lastMatch: when using PS, the match can be accessed using getLastMatch() afterwards

[ Low Level Mouse and Keyboard Actions ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

getMouseLocation()

Usage: Env.getMouseLocation() (a static method of Class Env).

returns: the current position of the mouse cursor as an object of Class Location.

[ Low Level Mouse and Keyboard Actions ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

keyDown( key | list of keys )

key | list of keys: one or more keys (use the constants of class Key). A list of keys is a concatenation using "+" of some constants.

The given keys are pressed and held until released by a later keyUp().

returns: the number 1 if the operation could be performed otherwise 0 (integer null).

[ Low Level Mouse and Keyboard Actions ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

keyUp( [ key | list of keys ] )

[ key | list of keys ]: one or more keys (use the constants of class Key). A list of keys is a concatenation using "+" of some constants.

The given keys are released. If no key is given, all currently pressed keys are released.

returns: the number 1 if the operation could be performed otherwise 0 (integer null).

[ Low Level Mouse and Keyboard Actions ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

5.8. Exception FindFailed

As a default, find operations (explicit and implicit) when not successful raise an exception FindFailed, that will stop the script immediately. This is of great help, when developing a script, to step by step adjust timing and similarity. When the script runs perfectly, then an exception FindFailed signals, that something is not as it should be.

To implement some checkpoints, where you want to asure your workflow, use exists(), that reports a not found situation without raising FindFailed (returns False instead).

To run all or only parts of your script without FindFailed exceptions to be raised, use setThrowException() to switch it on and off as needed.

For more sophisticated concepts, you can implement your own exception handling using the standard Python construct try: except:.

Example: 3 solutions for a case, where you want to decide how to proceed in a workflow based on the fact that a specific image can be found. (pass is the python statement, that does nothing, but maintains indentation to form the blocks) 

# --- nice and easy
if exists("path-to-image"): # no exception, returns None when not found
        pass # it is there
else:
        pass # we miss it

# --- using exception handling
# every not found in the try block will switch to the except block
try:
        find("path-to-image")
        pass # it is there
except FindFailed:
        pass # we miss it

# --- using setThrowException
setThrowException(False) # no exception raised, not found returns None
if find("path-to-image"):
        setThrowException(True) # reset to default
        pass # it is there
else:
        setThrowException(True) # reset to default
        pass # we miss it

[ Navigator ] - [ top of Document ]

setThrowException( False | True ) / getThrowException()

  • setThrowException( False | True )

By using this method you control, how Sikuli should handle not found situations. If used without specifying a region, the default/primary screen (default region SCREEN) is used.

True: from now on, unsuccessful find operations (explicit or implicit) will raise exception FindFailed (the default when a script is started).

False: from now on, unsuccessful find operations will not raise exception FindFailed. Find operations will return None, actions like click() will do nothing and mostly return 0.

  • getThrowException()

returns: the current setting as True or False (after start of script, this is True by default). If used without specifying a region, the default/primary screen (default region SCREEN) is used.

[ Exception FindFailed ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

5.9. Grouping Method Calls (with Region:)

instead of 

# reg is a region object
if not reg.exists(image1):
        reg.click(image2)
reg.wait(image3, 10)
reg.doubleClick(image4)

you can say 

# reg is a region object
with reg:
        if not exists(image1):
                click(image2)
        wait(image3, 10)
        doubleClick(image4)

All methods inside the with block that have the region omitted are redirected to the region object specified at the with statement.

[ Grouping Method Calls ] - [ Methods of Region ] - [ Navigator ] - [ top of Document ]

6. Class Screen

Table of Contents

Class Screen is there, to have a representation for a pysical monitor where the capturing process (grabbing a rectangle from a screenshot, to be used for further processing with find operations is implemented. For multi monitor environments it contains features to map to the relevant monitor.

Since Screen is of class Region, grouping method calls can be used with a screen object. Normally this should only be relevant in Multi Monitor Environments, to use it for other screens, than the default/primary screen, where you have this feature by default. Be aware, that using the whole screen for find operations may have an impact on performance. So if possible either use setROI() or restrict a find opeation to a smaller region object (e.g. reg.find()) to speed up processing.

[ Navigator ] - [ top of Document ]

6.1. Methods of Screen

Since class Screen extends class Region, all methods of class Region can be used with an existing screen object.

As a convenience here you have the most useful ones:

click - drag - dragDrop - dropAt - doubleClick - exists - find - findAll - getCenter - getH - getLastMatch - getLastMatches - getROI - getW - getX - getY - hover - keyDown - keyUp - mouseDown - mouseMove - mouseUp - observe - onAppear - onVanish - onChange - paste - rightClick - setAutoWaitTimeout - setH - setROI - setThrowException - setW - setX - setY - stopObserver - type - wait - waitVanish

additionally:

capture - getBounds - getNumberScreens - selectRegion

[ Navigator ] - [ top of Document ]

6.2. Screen: Setting, Getting Attributes and Information

getBounds - getNumberScreens

Screen(), Screen( id )

id: an integer number

It creates a new screen object, that represents the default/primary monitor (whose id is 0), if id is omitted. Numbers 1 and higher represent additional monitors that are available at the time, the script is running (read for details).

Using numbers, that do not represent an existing monitor, will stop the script with an error. So you may either use getNumberScreens() or exception handling, to avoid this.

returns: a new screen object.

Note: If you want to access the default/primary monitor ( Screen(0) ) without creating a new screen object, use the constant reference SCREEN, that is initiated when your script starts: SCREEN=Screen(0).

[ Screen: Setting and Getting Attributes ] - [ Methods of Screen ] - [ Navigator ] - [ top of Document ]

getNumberScreens()

returns: the number of screens available at the time the script is running (read for details).

[ Screen: Setting and Getting Attributes ] - [ Methods of Screen ] - [ Navigator ] - [ top of Document ]

getBounds()

returns: a rectangle object, where width and height denote the dimensions of the screen object representing a monitor. This attribute is taken from the operating system and cannot be changed using Sikuli script.

Note: In case of having more than one Monitor active, read Multi Monitor Environments for more information.

[ Screen: Setting and Getting Attributes ] - [ Methods of Screen ] - [ Navigator ] - [ top of Document ]

6.3. Screen as (Default) Region

Normally all region methods are used as reg.find(PS), where reg is a region object (or a screen or a match object). If written as find(PS) it acts on the default screen being the implicit region in this case. In Multi Monitor Environments this is Screen(0), that represents the primary monitor (use the constant reference SCREEN, to access it all the time).

So its a convenience feature, that can be seen as an implicit use of the python construct '''with object:'''.

On the other hand this may slow down processing speed, because of time consuming searches. So to speed up processing, saying region.find() will restrict the search to the specified rectangle. Another possibility is to say setROI() to restrict the search for all following find operations to a smaller region than the whole screen. This will speed up processing, if the region is significantly smaller than the whole screen.

Note: lastMatch and lastMatches of the default screen have to be accessed using SCREEN.getLastMatch() and SCREEN.getLastMatches(). So be careful, when using the default region concept.

[ Screen as (Default) Region ] - [ Methods of Screen ] - [ Navigator ] - [ top of Document ]

6.4. Capturing

capture - selectRegion

Capturing is the feature, that allows to grab a rectangle from a screenshot, to save it for later use. At each time, a capturing is initiated, a new screenshot is taken.

There are two different versions: the first one capture() saves the content of the selected rectangle in a file and returns its file name, whereas the second one selectRegion() just returns the position and dimension of the selected rectangle.

Both features are available in the IDE via buttons.

capture( [region | rectangle | text] ), capture( x, y, w, h )

  • no parameter: enters the interactiv mode
  • text: enters the interactiv mode and text is displayed for about 2 seconds in the middle of the screen
  • region: an existing region object
  • rectangle: an existing rectangle object (e.g. as a return value of another region method)
  • x, y, w, h: position (x,y) and dimension width and height of the rectangle to capture

Interactive mode: The script enters the screen-capture mode like when clicking the button in the IDE, enabling the user to capture a rectangle on the screen. If no text is given, the default "Select a region on the screen" is displayed.

If any arguments other than text are specified, capture() automatically captures the given rectangle of the screen. In any case, a new screenshot is taken, the content of the selected rectangle is saved in a temporary file. The file name is returned and can be used later in the script as a reference to this image. It can be used directly in cases, where a parameter PS is allowed (e.g. find(), click(), ...).

returns: the path to the file, where the captured image was saved. In interactive mode, the user may cancel the capturing, in which case None is returned.

[ Capturing ] - [ Methods of Screen ] - [ Navigator ] - [ top of Document ]

selectRegion()

Enables the user of a script to select a rectangle on the screen. For a description read here.

returns: a new region object or None, if the user cancels the capturing process.

[ Capturing ] - [ Methods of Screen ] - [ Navigator ] - [ top of Document ]

6.5. Multi Monitor Environments

If more than one monitor is available, Sikuli is able to manage regions and click points on these monitors.

The base is the coordinate system (picture above), that positions the primary monitor with its upper left corner at (0,0) extending the x-direction to the right and the y-direction towards the lower boundary of the screen. The position of additional monitors can be configured in the operating system to be on either side of the primary monitor, with different positions and sizes. So monitors left of the primary will have pixels with negative x-values and monitors above will have negative y-values (left and above both x and y are negative).

At script start, Sikuli gets the relevant information from the operating system and creates respective screen objects, that have an ID (0 for the primary monitor, 1 and higher for additional monitors with a maximum of one less than the number of screens) and know the rectangle, they cover in the coordinate system. These informations are readonly for a script.

These predefined screen objects can be accessed with Screen(0), Screen(1), ... and are normally used to create your own screen objects. The possibility to use the region methods on a default region mapped to the primary monitor is implemented with the constant reference SCREEN. This concept is only available for the primary monitor.

How to get the relevant information:

  • getNumberScreens() returns the number of available screens
  • getBounds() returns the rectangle, that is covered by the default/primary monitor
  • scr.getBounds() returns the rectangle, that is covered by the screen object scr, that was created before using Screen( id ).

You should analyse the information returned by this methods, to evaluate the specific situation with your monitor configuration.

So with its covered rectangle, a screen object is always identical with the monitor it was created based on Screen( id). Using setROI() to restrict the region of interest for find operations has no effect on the base rectangle of the screen object.

On the other hand region objects and location objects can be positioned anywhere in the coordinate system. Only when a find operation or a click action has to be performed, the objects rectangle or point has to be inside the rectangle of an existing monitor (basically repersented by Screen(0), Screen(1), ...). When this condition is met, everything works as expected and known from a single monitor system.

With finding and acting there are the following exceptions:

  • Point Outside: a click point is outside any monitor rectangle. The clickpoint will be mapped to the edges or corners of the primary monitor according to the relative position:
    • to the edges if its x or y value is in the range of the respective edge (right, left, above, below)
    • to the corners, if x and y are outside any range of any edge (left/above -> upper left corner, ...)
  • Region Outside: a region is completely outside any Monitor
    • a click action is handled according Point Outside
    • a find operation will always fail
  • Region Partly Outside: a region is partly outside of one monitor, no overlap on another monitor
    • a click action is handled according Point Outside, if this condition is met, otherwise it is processed
    • for a find operation, the region will be reduced to the part totally contained in the monitor rectangle
  • Region Overlaps Monitors: a region overlaps two or more monitors
    • a click action is handled according Point Outside, if this condition is met, otherwise it is processed
    • for a find operation, the region will be reduced to the part totally contained in the monitor rectangle belonging to the screen with the smallest id (so with 2 monitors this is always the primary monitor), so the find may fail, because a possible match is partly or totally outside the selected monitor.

An interactive capture (the user selects an image / a rectangle - capture() - selectRegion()) will automatically be restricted to the monitor rectangle, where it was started.

A scripted capture using a rectangle or a region, will be handled according to the exceptions above.

  • Region Outside no image captured, None is returned
  • Region Partly Outside the returned image will only cover the part contained in the monitor rectangle
  • Region Overlaps Monitors the returned image will only cover the part contained in the monitor rectangle of the monitor with the smallest id.

Based on the knowledge of your monitor configuration, you can now start some further evaluations using e.g. hover() together with setShowActions(True).

[ Multi Monitor Environments ] - [ Methods of Screen ] - [ Navigator ] - [ top of Document ]

7. Class Location

Table of Contents

This class is there as a convenience, to handle single points on the screen directly by its position (x, y). It iss mainly used in the actions on a region, to directly denote the click point. It contains methods, to "move" a point around on the screen.

[ Navigator ] - [ top of Document ]

7.1. Methods of Location

above - below - getsetL - getsetL - left - offset - right - getsetL - getsetL

[ Navigator ] - [ top of Document ]

7.2. Creating a Location, Setting and Getting Attributes

Location( x, y )

returns: a new location object, representing the position (x, y) on the screen.

[ Creating a Location, Setting and Getting Attributes ] - [ Methods of Location ] - [ Navigator ] - [ top of Document ]

getX(), getY(), setX( number ), setY( number )

getX(), getY(): the x or y value are returned (saying location.x or location.y is equivalent).

setX( number ), setY( number ): the respective attribute is set to number (saying location.x = number or location.y = number is equivalent).

[ Creating a Location, Setting and Getting Attributes ] - [ Methods of Location ] - [ Navigator ] - [ top of Document ]

offset( dx, dy )

returns: a new Location(x+dx, y+dy)

[ Creating a Location, Setting and Getting Attributes ] - [ Methods of Location ] - [ Navigator ] - [ top of Document ]

above( dy )

returns: a new Location(x, y-dy)

[ Creating a Location, Setting and Getting Attributes ] - [ Methods of Location ] - [ Navigator ] - [ top of Document ]

below( dy )

belowL

returns: a new Location(x, y+dy)

[ Creating a Location, Setting and Getting Attributes ] - [ Methods of Location ] - [ Navigator ] - [ top of Document ]

left( dx )

returns: a new Location(x-dx, y)

[ Creating a Location, Setting and Getting Attributes ] - [ Methods of Location ] - [ Navigator ] - [ top of Document ]

right( dx )

returns: a new Location(x+dx, y)

[ Creating a Location, Setting and Getting Attributes ] - [ Methods of Location ] - [ Navigator ] - [ top of Document ]

8. Class Match

Table of Contents

An object of class Match represents the result of a successful find operation. It has the rectangle dimension of the image, that was used to search. It knows the point of its upper left corner on an existing monitor, where it was found. You can act on it using the applicable methods of Class Region.

[ Navigator ] - [ top of Document ]

8.1. Methods of Match

Since class Match extends class Region, all methods of class Region can be used with a match object.

As a convenience here you have the most useful ones:

above - below - click - drag - dragDrop - dropAt - doubleClick - exists - find - findAll - getCenter - getH - getLastMatch - getLastMatches - getScreen - getW - getX - getY - hover - inside - left - mouseMove - nearby - observe - onAppear - onVanish - onChange - paste - right - rightClick - selectRegion - setAutoWaitTimeout - setThrowException - stopObserver - type - wait - waitVanish

additionally available:

getScore - getTarget

[ Navigator ] - [ top of Document ]

8.2. Creating a Match, Getting Attributes

A match object is created as the result of an explicit find operation. It can be saved in a variable for later use with actions like click().

It has the rectangle dimension of the image, that was used to search. It knows the point of its upper left corner on an existing monitor, where it was found. It knows the similarity it was found with and a click point to be used, if set by a pattern. 

m = find("path-to-image-file") # m is a reference to a match object, if found
print m # message area: Match[833,248 447x141] score=0.89, target=null

m = find(Pattern("path-to-image-file").similar(0.5).targetOffset(100,0)) # m is a reference to a match object, if found
print m # message area: Match[406,461 140x123] score=0.75, target=(576,522)

For all other aspects the features and attributes of class Region apply.

[ Creating a Match, Getting Attributes ] - [ Methods of Match ] - [ Navigator ] - [ top of Document ]

getScore()

returns: the similarity that the image/pattern was found with as a decimal value: 0 < value <= 1

[ Creating a Match, Getting Attributes ] - [ Methods of Match ] - [ Navigator ] - [ top of Document ]

getTarget()

returns: a location object, that will be used as click point with actions like click(). If the find operation was based on a pattern without using targetOffset() or an image, this will be the same as getCenter(). Otherwise it will be calculated and returned as getCenter().offset(targetX, targetY), where targetX and targetY are the respective values of targetOffset() of the pattern used for the find operation.

[ Creating a Match, Getting Attributes ] - [ Methods of Match ] - [ Navigator ] - [ top of Document ]

8.3. Iterating over Matches after findAll()

A find operation findAll() returns an iterator object that contains all matches found. It can be stepped through to get each match object in turn. Remember, that a reference to the iterator of matches is stored in the respective region and can be accessed using getLastMatches().

Important to know:

  • per definition, an iterator can be stepped through only once - it is empty afterwards
  • it has to be destroyed manually using iteratorMatches.destroy(), when used with for: or while:
  • when used in a with: construct, it is destroyed automatically

With class Finder you can read more about the basics of an iterator. There you find an example, how to save the contained matches for later use.

The methods to use:

  • hasNext(): returns True, if there is at least one match left, otherwise False
  • next(): returns the next match, if there is at least one match left, otherwise None
  • destroy(): destroys the iterator object (releases memory)

Example: using while: with default screen 

findAll("path-to-an-imagefile") # find all matches
mm = SCREEN.getLastMatches()
while mm.hasNext(): # loop as long there is a first and more matches
        print "found: ",  mm.next() # access the next match in the row

print mm.hasNext() # is False, because mm is empty now
print mm.next() # is None, because mm is empty now
print SCREEN.getLastMatches().hasNext() # is False also ;-)
mm.destroy() # to save memory

Example: using with: with default screen 

with findAll("path-to-an-imagefile") as mm:
        while mm.hasNext(): # loop as long there is a first and more matches
                print "found: ",  mm.next() # access the next match in the row
# mm will be None afterwards (destroyed automatically)

[ Iterating over Matches ] - [ Methods of Match ] - [ Navigator ] - [ top of Document ]

9. Class Finder

Table of Contents

A Finder object is based on an iterator of matches and allows to search for a visual object in an image file that you provide (e.g. a screenshot taken and saved in a file before). After setting up the finder object and doing a find operation, you can iterate through the found matches if any.

Important to know:

  • per definition, an iterator can be stepped through only once - it is empty afterwards
  • it has to be destroyed manually using iteratorMatches.destroy(), especially when used with for: or while:
  • when used in a with: construct, it is destroyed automatically

Compared with the region based find operation findAll(), no exception FindFailed is raised in case of nothing found at all (use haseNext() to check) and what region.lastMatches is with findAll() is the finder object itself here.

Note: With this version, there is no chance, to get the number of matches in advance. If you would iterate through to count, afterwards your finder would be empty. So in this case, you have to save your matches somehow (one possible solution see example below).

[ Navigator ] - [ top of Document ]

9.1. Methods of Finder

find - hasNext - next

[ Navigator ] - [ top of Document ]

9.2. Creating a Finder

Finder( path-to-imagefile )

path-to-imagefile: just what it says (e.g. your screenshot)

First you have to create a new finder object.

9.3. Using a Finder

The workflow always is, that first do a find operation and afterwards go through the matches found. After a complete iteration, the finder object is empty again. So you could start a new find operation again.

find( path-to-imagefile, [ similarity ] )

path-to-imagefile: just what it says (your image to search for)

[ similarity ]: the minimum similarity a match should have, if omitted, the default is used (read for details)

Searches for the given image with reference to the minimum similarity and stores the matches in the finder object.

hasNext()

Checks if there are more matches available. So when used the first time after a find(), it tells you wether there are matches at all.

returns: True if there are more matches available, otherwise False

next()

Accesses the next match in the row. The returned reference to a match object is no longer available in the finder object afterwards. So if you need it later on, you have to save it somehow.

returns: a match object

Example: just the basics 

f = Finder("path-to-an-imagefile") # create a Finder with your saved screenshot
img= "path-to-an-imagefile" # the image you are searching
f.find(img) # find all matches
while f.hasNext(): # loop as long there is a first and more matches
        print "found: ",  f.next() # access the next match in the row
print f.hasNext() # is False, because f is empty now
f.destroy() # to save memory (especially in loops when new objects are created all the time)

Example: we want to know the number of matches before 

f = Finder("path-to-an-imagefile") # create a Finder with your saved screenshot
img = "path-to-an-imagefile" # the image you are searching
f.find(img) # find all matches
mm = [] # an empty list
while f.hasNext(): # loop as long there is a first and more matches
        mm.append( f.next() ) # access next match and add to mm
print f.hasNext() # is False, because f is empty now
# now we have our matches saved in list mm
print len( mm ) # the number of matches
# we want to use our matches
for m in mm:
        print m # or what ever you want
f.destroy() # to save memory (especially in loops when new objects are created all the time)

[ Navigator ] - [ top of Document ]

10. Class Pattern

A pattern is used, to associate an image file with additional attributes used in find operations and when acting on a match object.

Minimum similarity: When using just an image file in a find operation, the search will be successful and return a match object, if the similarity of a possible match is 0.7 or higher. With a pattern object, you can associate a specific similarity value, that will be used as the minimum value, when this pattern object is searched (similar(), exact()). The IDE supports adjusting the minimum similarity with captured images (internally in the script, the images are turned into a pattern object definition)

click point: normally when clicking on a match, the center pixel of the associated rectangle is used. With a pattern object, you can define a different click point (targetOffset()).

[ Navigator ] - [ top of Document ]

10.1. Methods of Pattern

exact - getFilename - getTargetOffset - similar - targetOffset

[ Navigator ] - [ top of Document ]

10.2. Creating a Pattern, Setting and Getting Attributes

Pattern( string )

string: a path to an image file

This will initialize a new pattern object without any additional attributes. As long as no pattern methods are used additionally, it is the same as just using the image file name itself in the find operation.

returns: a new pattern object

similar( similarity )

similarity: the minimum similarity to use in a find operation as a decimal value: 0 < similarity <= 1

A new Pattern object is created containing the same attributes (image, click point) with the minimum similarity set to the specified value.

returns: a new pattern object

exact()

A new Pattern object is created containing the same attributes (image, click point) with the minimum similarity set to 1.0, which means exact match required.

returns: a new pattern object

targetOffset( dx, dy)

dx, dy: numbers without a fraction that can be negativ.

A new Pattern object is created containing the same attributes (image, similarity), but a different definition for the click point to be used, when acting on the match object returned by a successful find operation. It will be calculated as match.getCenter().offset(dx, dy) (details)

returns: a new pattern object

[ Creating a Pattern, Setting and Getting Attributes ] - [ Methods of Pattern ] - [ Navigator ] - [ top of Document ]

getFilename()

returns: the filename of the image, that is contained in the pattern object (as a string)

[ Creating a Pattern, Setting and Getting Attributes ] - [ Methods of Pattern ] - [ Navigator ] - [ top of Document ]

getTargetOffset()

returns: the currently specified target offset of the pattern object (as a location object)

[ Creating a Pattern, Setting and Getting Attributes ] - [ Methods of Pattern ] - [ Navigator ] - [ top of Document ]

11. Class VDict

Table of Contents

VDict implements a visual dictionary that has Python's conventional dictionary interfaces and whose syntax is modeled after that (thereafter called VDict)

Using a VDict a user can easily automate the tasks of saving and retrieving arbitrary data objects by using images as keys. The syntax of a VDict is modeled after that of the built-in Python dictionary data type. But be aware, that only operators and built-in methods of a Python dictionary are supported, when mentioned here.

Internally, a VDict is based on a dictionary with keys representing images using a path to an image file (so it is a string). The values, that are paired with the keys (together they form an item), can be any data type allowed in the Sikuli environment based on the Python language. To locate an item in a VDict, we use the same features as when searching for images in a region (so to match a key value, not the string representing a filename is used to compare, but the visual content applying a given similarity). If not yet familiar with, read for details beginning with the class Pattern.

When using a VDict, the images referenced by the keys must be available (read: bundel-path and path to an image file) [ Navigator ] - [ top of Document ]

11.1. Methods/Operators/Constants of VDict

vdict[ ] - vdict[ ] = - [ not in vdict] - del vdict[ ] - get -get1 - get_exact - len( ) - keys

11.2. Setting it up and getting Information

VDict( [ vdict | dict ] )

[ vdict | dict ]: another existing VDict named vdict in this case or a basic python dictionary named dict in this case, whose keys are strings containing image filenames, that can be located at runtime. If omitted, the new VDict is empty.

Constructs a new visual dictionary with the same mapping as the given dict.

returns: the new VDict

len( vdict )

vdict: an exisiting VDict named vdict in this case

returns: the number of keys in this visual dictionary.

keys()

returns: a list containing the keys of the VDict.

11.3. Managing Items

vdict[ path-to-an-imagefile ] = value

vdict: the name of an existing VDict object

path-to-imagefile: just what it says (your image to search for in the keys)

value: an arbitrary value

A new item is created, mappping the specified image to the specified value.

del vdict[ path-to-an-imagefile ]

vdict: the name of an existing VDict object

path-to-imagefile: just what it says (your image to search for in the keys)

Deletes the key and its corresponding value from this VDict.

11.4. Searching for specific Items

#SearchVDict

path-to-an-imagefile [not] in vdict

vdict: the name of an existing VDict object

path-to-imagefile: just what it says (your image to search for in the keys)

Tests if the image matches with at least one of the keys using the default similarity.

returns: True if at least one matches, False otherwise

vdict[ path-to-an-imagefile ]

vdict: the name of an existing VDict object

path-to-imagefile: just what it says (your image to search for in the keys)

returns: a list of all values to which the specified image matches with the default similarity.

get_exact( path-to-an-imagefile )

path-to-imagefile: just what it says (your image to search for in the keys)

returns: the one value to which the specified key is exactly matched.

get1( path-to-an-imagefile', similarity)

path-to-imagefile: just what it says (your image to search for in the keys)

returns: the one value to which the specified image is best matched with the given similarity.

get( path-to-an-imagefile', similarity, number )

path-to-imagefile: just what it says (your image to search for in the keys)

similarity: the similarity for matching (a decimal number between 0 and 1) number: maximum number of returned items.

returns: a list of the values, whose length can be limited by number, to which the specified image matches with the given similarity used as keys in the VDict.

11.5. Constants of VDict

_DEFAULT_SIMILARITY the default similarity for fuzzy matching. The range of this is from 0 to 1.0, where 0 matches everything and 1.0 does exactly matching. The default similarity is 0.7.

[ Navigator ] - [ top of Document ]

12. Key Constants

Table of Contents

Applicable usage situations for these predefined constants of special keys and key modifiers can be found in Acting on a Region and Low Level Mouse and Keyboard Actions.

[ Navigator ] - [ top of Document ]

12.1. Key Modifiers

Methods where key modifiers can be used:

click - dragDrop - doubleClick - rightClick - type

  • the oldies but goldies

KEY_ALT - KEY_CTRL - KEY_SHIFT

  • system specific Win/Mac

KEY_WIN ( Windows key ) - KEY_CMD (The Apple command key) - KEY_META (a synonym for KEY_WIN or KEY_CMD on the respective system)

Note: These constants are mapped to the according constants of the Java environment in the class java.awt.event.InputEvent. They should be used only as the parameter of the functions that have a modifiers parameter, e.g. type(), click(), etc, but not as a key parameter in keyDown() and keyUp().

[ Key Modifiers ] [ Key Constants ] [ Navigator ] - [ top of Document ]

12.2. Special Keys

The methods supporting the use of special keys are type(), keyDown(), and keyUp().

Usage: Key.CONSTANT (where CONSTANT is one of the following key names). Concatenation with strings with "+" can be used.

  • miscellanous keys

ENTER - TAB - ESC - BACKSPACE - DELETE - INSERT

  • function keys

F1 - F2 - F3 - F4 - F5 - F6 - F7 - F8 - F9 - F10 - F11 - F12 - F13 - F14 - F15

  • navigation keys

HOME - END - LEFT - RIGHT - DOWN - UP - PAGE_DOWN - PAGE_UP

  • key modifiers

ALT - CMD - CTRL - META - SHIFT - WIN

Note: The key modifier constants can not be used as the parameter of the functions that have a modifiers parameter, e.g. type(), click(), etc. They can be used only as a key parameter in keyDown() and keyUp().

[ Special Keys ] [ Key Constants ] [ Navigator ] - [ top of Document ]

13. Class Env

Though formally defined in this class, the description of the following methods is found in the applicable context:

getOS() returns information about the system, the script is running on ( -> General Information and Settings )

getOSVersion() returns the system's version number, the script is running on ( -> General Information and Settings )

getMouseLocation() returns the current location of the mouse pointer ( -> Low Level Mouse and Keyboard Actions )

getClipboard() returns the content of the Clipboard if it is text, otherwise an empty string. ( -> General Information and Settings )

[ Navigator ] - [ top of Document ]

Attachments