Thursday, July 18, 2013

Android game automation - part 1

First: this is borderline immoral so don't ask for any source code or help.


My friend got me into a repetitive Android game that I will not name here. Basically it's a different kind of Farmville (I assume) that requires you to mindlessly click 'animals' to 'farm' money from them. On top of that you have to also activate two type of farms in order to feed the animals and evolve them. Feeding is not a requirement, so it will only be done in the second iteration of this automation.

As a rule of thumb any task that takes you at least 5 minutes every day for a year should be automated if it could be done in less than 20 hours.



I was opening the app about 1-3 times a day, clicking on each animal and closing it. Even though it's mindless and almost fun it still took about 3 minutes each: 1 minute the log-in and load time, 1 minute the mindless task, 1 minute reorganizing stuff to have it perform better. There is clearly room for improvement because now the 'robot' logs in every 5 minutes, does the job and gets out. Initial estimation shows it earns about 80% more virtual money than I was able (wanting) to.


Organization

The playing area is composed of several rectangles that you have to swipe through. Above you see only one are but my current field is about 5x5 this.
  • The small houses with the gold floating on top of them should be clicked every 5 minutes or so.
  • The animal in the top right part should be clicked evey 2 hours or so.
  • The green farm bottom center should be activated every 10 hours or so by clicking on the Activate button a few times and then on the farm itself.

Reverse-engineering

The app is a crappy port, probably from iOS, so most if its part is native. It also uses a custom game engine that I haven't been able to find any info on. The game logic itself is stored in some binary files that I assume to be some recompiled Lua scripts.
Seeing that this would take too long I went down the brute-force approach. It would also work if the app were to be updated, which in this case happens at around two weeks. That's right, they are pushing 80-150MB every two weeks to every device in the user base!

Idea

I envisioned the initial implementation to just follow my moves. It is close to impossible to find a macro recorder for Android so I rolled my own.
Since the app was native it was not possible to click by finding out objects inside the Window Manager, it must be done via the low-level [hardware] event manager.

First, listing input devices:

root@android:/ # getevent -p
getevent -p
add device 1: /dev/input/event6
  name:     "HPTouchpad"
  events:
    ABS (0003): 0030  : value 0, min 0, max 0, fuzz 0, flat 0, resolution 0
                0035  : value 0, min 0, max 1024, fuzz 2, flat 0, resolution 0
                0036  : value 0, min 0, max 768, fuzz 1, flat 0, resolution 0
                0039  : value 0, min 0, max 0, fuzz 0, flat 0, resolution 0
  input props:
    <none>
add device 2: /dev/input/event1
  name:     "pmic8058_pwrkey"
  events:
    KEY (0001): 006b  0074
  input props:
    <none>
add device 3: /dev/input/event2
  name:     "lsm303dlh_mag_sysfs"
  events:
    ABS (0003): 0000  : value -434, min -8100, max 8100, fuzz 0, flat 0, resolution 0
                0001  : value 848, min -8100, max 8100, fuzz 0, flat 0, resolution 0
                0002  : value -950, min -8100, max 8100, fuzz 0, flat 0, resolution 0
  input props:
    <none>
add device 4: /dev/input/event3
  name:     "lsm303dlh_acc_sysfs"
  events:
    ABS (0003): 0000  : value -19, min -8000, max 8000, fuzz 0, flat 0, resolution 0
                0001  : value -12, min -8000, max 8000, fuzz 0, flat 0, resolution 0
                0002  : value 1029, min -8000, max 8000, fuzz 0, flat 0, resolution 0
                0008  : value 0, min -2147483648, max 2147483647, fuzz 0, flat 0, resolution 0
                0028  : value 0, min -2147483648, max 2147483647, fuzz 0, flat 0, resolution 0
  input props:
    <none>
add device 5: /dev/input/event4
  name:     "isl29023 light sensor"
  events:
    ABS (0003): 0028  : value 12, min 0, max 798400, fuzz 0, flat 0, resolution 0
  input props:
    <none>
add device 6: /dev/input/event5
  name:     "headset"
  events:
    KEY (0001): 00a4
    SW  (0005): 0002  0004
  input props:
    <none>
add device 7: /dev/input/event0
  name:     "gpio-keys"
  events:
    KEY (0001): 0072  0073  00e8  00f9
  input props:
    <none>
Just by name we can figure out that event6 events are pushed by the touchscreen, event1 is the power key, event2 is the magnetometer, event3 the accelerometer, event4 the light sensor, event5 the headphone jack detection and event0 the remaining hardware keys (volume buttons, AFAIK).

Recording the events

it's just a matter of running getevent either on the device itself or via adb. Something like "getevent >events.out". The output should be similar to:
/dev/inputevent3: 0003 0000 fffffff1
/dev/inputevent3: 0003 0001 ffffffeb
/dev/inputevent3: 0003 0002 000003fd
/dev/inputevent3: 0000 0000 00000000
/dev/inputevent3: 0003 0000 fffffffb
/dev/inputevent3: 0003 0001 fffffff2
/dev/inputevent3: 0003 0002 000003fd
/dev/inputevent3: 0000 0000 00000000
/dev/inputevent6: 0003 0030 00000032/dev/inputevent6: 0003 0035 00000217/dev/inputevent6: 0003 0036 000000b8/dev/inputevent6: 0000 0002 00000000/dev/inputevent6: 0000 0000 00000000/dev/inputevent3: 0003 0000 fffffff9/dev/inputevent3: 0003 0001 fffffff0/dev/inputevent3: 0003 0002 000003f8/dev/inputevent3: 0000 0000 00000000/dev/inputevent6: 0003 0039 00001988/dev/inputevent6: 0003 0030 00000032/dev/inputevent6: 0003 0035 00000217/dev/inputevent6: 0003 0036 000000b8/dev/inputevent6: 0000 0002 00000000/dev/inputevent6: 0000 0000 00000000/dev/inputevent6: 0003 0039 00001988   
You might spot a problem here that I did not recognize in the beginning. Anyway, onward to

Playing back the events:

During some kind of twisted design decision it was decided that the "sendevent" command has a different format than what "getevent" provides. Luckily I found a blog post that had an almost working solution written in Python that did everything:

The problems

Everything went nice but for a recorded session lasting about 30 seconds the script took about 15 minutes to replay.
Also, all the micro-swipes were being converted to clicks and getting lost and all the swipe inertia was being lost because the speed was much lower. You normally don't realize all this stuff happening behind the scenes this when using a capacitive touchscreen.

The first issue could be dealt with by doing a few optimizations. You can see from the listing above that for every touch event there is are a ton of accelerometer events. A simple grep takes care of that.
Also, sending events from the PC to the phone is really slow so it can be solved like this:
:start@time /tcall monkeyrunner.bat %CD%\mainscreen.pyadb push mainscreen.scr /sdcard/mainscreen.scradb shell sh /sdcard/mainscreen.scr@time /ttimeout /t 30goto startpause
Where the python script just unlocks the screen, kills the app if it is running and starts it again. The mainscreen.scr file contains the sendevent commands line by line:
#!/bin/sh
echo Running - drawing function
sendevent /dev/input/event6 3 53 663
sendevent /dev/input/event6 3 54 116
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 663
sendevent /dev/input/event6 3 54 116
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 663
sendevent /dev/input/event6 3 54 116
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 657
sendevent /dev/input/event6 3 54 127
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 652
sendevent /dev/input/event6 3 54 138
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 645

Not all events are needed so some of them were stripped, I forgot which, but probably touch area and several of the touch confirmations. Here's a draft I had in the folder, ending with a parsed getevent.
Multi-touch devices use the following Linux input events:
ABS_MT_POSITION_X: (REQUIRED) Reports the X coordinate of the tool.
ABS_MT_POSITION_Y: (REQUIRED) Reports the Y coordinate of the tool.
ABS_MT_PRESSURE: (optional) Reports the physical pressure applied to the tip of the tool or the signal strength of the touch contact.
ABS_MT_TOUCH_MAJOR: (optional) Reports the cross-sectional area of the touch contact, or the length of the longer dimension of the touch contact.
ABS_MT_TOUCH_MINOR: (optional) Reports the length of the shorter dimension of the touch contact. This axis should not be used if ABS_MT_TOUCH_MAJOR is reporting an area measurement.
ABS_MT_WIDTH_MAJOR: (optional) Reports the cross-sectional area of the tool itself, or the length of the longer dimension of the tool itself. This axis should not be used if the dimensions of the tool itself are unknown.
ABS_MT_WIDTH_MINOR: (optional) Reports the length of the shorter dimension of the tool itself. This axis should not be used if ABS_MT_WIDTH_MAJOR is reporting an area measurement or if the dimensions of the tool itself are unknown.
ABS_MT_ORIENTATION: (optional) Reports the orientation of the tool.
ABS_MT_DISTANCE: (optional) Reports the distance of the tool from the surface of the touch device.
ABS_MT_TOOL_TYPE: (optional) Reports the tool type as MT_TOOL_FINGER or MT_TOOL_PEN.
ABS_MT_TRACKING_ID: (optional) Reports the tracking id of the tool. The tracking id is an arbitrary non-negative integer that is used to identify and track each tool independently when multiple tools are active. For example, when multiple fingers are touching the device, each finger should be assigned a distinct tracking id that is used as long as the finger remains in contact. Tracking ids may be reused when their associated tools move out of range.
ABS_MT_SLOT: (optional) Reports the slot id of the tool, when using the Linux multi-touch protocol 'B'. Refer to the Linux multi-touch protocol documentation for more details.
BTN_TOUCH: (REQUIRED) Indicates whether the tool is touching the device.
BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_BACK, BTN_SIDE, BTN_FORWARD, BTN_EXTRA, BTN_STYLUS, BTN_STYLUS2: (optional) Reports button states.
BTN_TOOL_FINGER, BTN_TOOL_PEN, BTN_TOOL_RUBBER, BTN_TOOL_BRUSH, BTN_TOOL_PENCIL, BTN_TOOL_AIRBRUSH, BTN_TOOL_MOUSE, BTN_TOOL_LENS, BTN_TOOL_DOUBLETAP, BTN_TOOL_TRIPLETAP, BTN_TOOL_QUADTAP: (optional) Reports the tool type.

[   71418.045740] EV_ABS       ABS_MT_TRACKING_ID   0000215d[   71418.045789] EV_ABS       ABS_MT_TOUCH_MAJOR   00000019[   71418.045813] EV_ABS       ABS_MT_POSITION_X    0000026a[   71418.045837] EV_ABS       ABS_MT_POSITION_Y    0000017a[   71418.045861] EV_SYN       SYN_MT_REPORT        00000000[   71418.045882] EV_SYN       SYN_REPORT           00000000[   71418.056783] EV_ABS       ABS_MT_TRACKING_ID   0000215d[   71418.056838] EV_ABS       ABS_MT_TOUCH_MAJOR   00000019[   71418.056862] EV_ABS       ABS_MT_POSITION_X    0000026a[   71418.056885] EV_ABS       ABS_MT_POSITION_Y    0000017a[   71418.056908] EV_SYN       SYN_MT_REPORT        00000000[   71418.056929] EV_SYN       SYN_REPORT           00000000[   71418.067805] EV_ABS       ABS_MT_TRACKING_ID   0000215d

All these changes improved the time from 15 minutes down to 5 minutes, but it was not good enough for 'production'.

But the most important problem of this approach was that it had no provision for application layout update and required a new recording with each game field change (i.e. moving animals around). Recording including parsing and everything took around 2-5 minutes per session and was probably required every two days.

No comments:

Post a Comment