What
After being introduced to some of the games on Omgpop, the clear next step was to automate them for fun. This first video shows Swapples, a Bejeweled-like game. The board is screengrabbed then the tiles are visually recognized by an SVM. The first half of the video shows the program running when the board is allowed to “settle” (differencing subsequent frames gives <1% pixels changed). The second half of the video shows the program running without waiting for the board to settle, illustrating how much time is spent waiting and how fast it could (ideally) make moves. I’m hoping to return to this project at some point to make it smarter about choosing moves (prioritize 4-/5-in-a-row, hourglasses, multipliers) and to make moves more quickly (make multiple moves per analyzed screengrab by tracking how a move will affect the board).
This second video shows Letterblox, which I partially-automated by traversing a trie of English words based on user-input letters. The trie was a nice exercise in recursion, but it turned out that asking a plain dictionary/hashtable for all permutations of the letters was faster.
This third video shows Typow, which I automated by screengrabbing, OCRing, then playing back the text.
Tech
- Swapples - Python with scikits-learn SVMs for detecting tiles, PIL for screen grabbing, numpy, and SendKeys for sending clicks
- Letterblox - Python with SendKeys for sending keystrokes
- Typow - AutoHotkey script that uses ImageMagick for screen grabbing and preprocessing, and Tesseract to OCR the image