Showcase


6
May 10

Use Clipboard to Extract Text with Sikuli

Sikuli 0.10 has added tons of cool new features. Now it is easier to write smarter scripts to do more interesting and useful tasks than you can ever imagine. Here is one of my favorites. The original idea of this script was contributed by RaiMan, one of the most active Sikuli users. I simplified it a little bit and made a video to show how it runs in a real environment.

The source code of the Sikuli script that downloads a bunch of files.

The script to download a bunch of files.

The goal of this script is to download all ZIP files whose file name starts with “Sikuli-IDE-osx” in a web page. To find all ZIP files is easy, because they have an identifiable icon. However, we can’t simply click on the icon. We need to click on the hyperlink next to the icon, instead. The really tricky thing is, how to know a hyperlink that starts with “Sikuli-IDE-osx”?

The simplest solution is capturing a screenshot of the text “Sikuli-IDE-osx”, and putting the screenshot as the parameter of click(). Unfortunately, this way is not really robust since the images of text only have tiny differences. Sikuli’s fuzzy image matching engine may treat “S1kul1-1DE-esx” as the same as what we want. So, a right way to do this is – extracting the text from the web page.

You may recall that there was an ancient technology named OCR (Optical Character Recognition), which recognizes text from scanned books. However, current OCR engines do not really work well. They are slow and inaccurate.

In fact, even without OCR, we still can extract text from the screen using Sikuli.

How?

Use the clipboard!

The following script tells you how to do that. At first, we use Sikuli to find all locations of the ZIP icon (line 7). On line 8, we sort the list of the matched locations by their y coordinate, it’s not necessary though. The key steps are line 10 and 11. We click on the space between the icon and the hyperlink next to it using the new target offset attribute of a pattern. (denoted by a small red cross in the ZIP icon on line 7. See the figure below for its setting window.)

And then we hold the mouse button and drag to the location 600-pixel right to the center of the icon. This step (line 2) actually simulates using a mouse to select a line of text with a dragDrop(). After the selection, we simply press Command+C (Ctrl-C on Windows) to copy the selected text into the system clipboard.

Copy (extract) text from a browser.

Aha! We got the precise text in the clipboard. :)

The rest of the script are trivial. Sikuli 0.10 provides a new API Env.getClipboard() to get the content of the system clipboard. We can use this function to get the text of the hyperlinks and then use Python’s string function (startswith) to filter out the files we need.

I also made a tiny video that shows how this script looks like when it is running.


2
Mar 10

Sikuli plays a piano

Sikuli is not only visual but musical. ram B has uncovered Sikuli’s musical potential with a creative script that plays a virtual piano. The script opens a browser and opens up an online virtual piano. Then it asks for an input to choose from a collection of two songs: Terminator’s theme and Godfather’s theme.

The whole script is shown below. You can also download the script to try it out on your computer. Enjoy!!


1
Mar 10

Sikuli signs in Twitter and tweets

A while ago we posted several examples of automatic tweets on Twitter sent by Sikuli. Andrew Burton (a.k.a. jarod), from whom one of these tweets were originated, sent us a Twitter script he created that does much more than the simple example we showed last time. This script not only sends a tweet but also signs in one’s Twitter account. When the script is run, it prompts the user for signin id, password, and message. Then the script goes about looking for various buttons to click and text fields to type. In the process, it starts a web browser, types the address of Twitter, waits for the Twitter logo to appear, clicks on the sign-in button, types the login and password, clicks the message box, types the message, and finally clicks the update button to submit the tweet.

The whole script is shown below.


28
Feb 10

Sikuli searches Picasa photos and starts a slideshow

Many thanks to ram B for contributing another interesting script. The purpose of the script is to search for pictures on Picasa. The script first opens Picasa in Firefox browser. Then in this example it enters a keyword “cars” and clicks on the “search” button to submit the keyword. Unlike Flickr search, in Picasa, one has to click the “play” button even after clicking the slide show button. So this script will start the slide show increase the speed by 1 sec, hit the play button and will close the window after displaying around 10 pictures.

The full script can be seen below.


27
Feb 10

Sikuli automates FFXI login

FFXI (i.e., Final Fantasy 11) is one of the most popular online RPG games, but notorious for its ridiculously long and tedious login process. firewing at the Bluegarttr created a Sikuli script to automate this process. By exporting the script to an executble (.skl) file and placing a shortcut on the desktop, all one needs to do to start playing FFXI is to double-click the shortcut. How easy!!

The full script annotated by the author is shown below.


26
Feb 10

Sikuli plays Beatles radio

Today’s showcase script is contributed by Ram B for tuning in to an online radio station on last.fm to listen to Beatles music. This script launches Firefox, types “www.last.fm” in the address bar, types “beatles” in the search box, clicks the “GO” button to confirm the search, and finally clicks on “Play The Beatles Radio” button to play. The entire script can be viewed below.


17
Feb 10

Sikuli plays Mafia Wars

Sikuli is fun! Today’s showcase script is contributed by Garrett Bartley. He uses Sikuli to write a script to play Mafia Wars online. The script automates a simple yet important task in Mafia Wars: upgrading the casino!!

To display data, Sikuli currently supports only popup messages and debug messages, which can be very limiting. Garrett uses a very clever trick to get around this limitation. He uses Sikuli to open up another text editor and type the data in the editor to display.

It may not be clear to many Sikuli users that under the hood of Sikuli, there is a Java virtual machine providing a rich set of UI components such as tables and lists that can be called upon to display data. In the future release of Sikuli, we plan to introduce simple API calls to uses these components for output, while hiding the complexity of Java programming. Please stay tuned!


16
Feb 10

Sikuli is automated by another tool to automate itself

We thought one thing Sikuli can not automate is itself, since it hides itself every time when it executes a script. We are proven wrong by Koichi Tamura, who has demonstrated it is possible to run a command-line Sikuli script to click on the buttons of another Sikuli IDE in the foreground in order to run other Sikuli scripts. More interestingly, the process of loading Sikuli scripts into the IDE is automated by yet another automation tool. This tool is a visual programming tool Koichi is currently developing himself. Below is the example demonstrated by Koichi. Two Sikuli scripts are represented as two nodes connected by an arrow. One script opens a “hello world” popup, while the other opens a new tab in a web browser and loads his blog in the tab.

Watch the video in YouTube to see this example in action:


12
Feb 10

Sikuli sends and checks tweets

One reason Sikuli is quickly growing in popularity is that people write tweets about it. Some people even use Sikuli to send tweets. This only takes three lines of code: click on the text area, type some message, and click on the update button:

Below are some examples of Sikuli tweets (thanks to Hardmood and Jarod).

Sikuli can also be used to check Twitter updates. This script is created by Mike Damage for checking if his friend Steve has recently sent a new tweet. It sits in a loop to wait for the portrait photo of Steve to appears, and when it does, displays a notification message that Steven has a new tweet!

Below is the Sikuli script for checking tweet updates:


11
Feb 10

Sikuli cleans

Many people have found Sikuli useful for automating daily maintenance tasks such as clearing the content of the recycle bin and making backups, especially for those people who only know how to perform these tasks by clicking through the UI. Using Sikuli, it is quite easy and intuitive to create a script to automate tedious tasks like these.

The script we would like to showcase today is created by Jon Atkinson. The purpose of this script is to run system clean. The script launches a scanner application, starts scanning, waits for the scanning to complete, and presses the remove button to confirm the removal of unwanted files. The details of the script is shown below: