First: this is borderline immoral so don't ask for any source code or help.
My friend got me into a repetitive Android game that I will not name here. Basically it's a different kind of Farmville (I assume) that requires you to mindlessly click 'animals' to 'farm' money from them. On top of that you have to also activate two type of farms in order to feed the animals and evolve them. Feeding is not a requirement, so it will only be done in the second iteration of this automation.
As a rule of thumb any task that takes you at least 5 minutes every day for a year should be automated if it could be done in less than 20 hours.
I was opening the app about 1-3 times a day, clicking on each animal and closing it. Even though it's mindless and almost fun it still took about 3 minutes each: 1 minute the log-in and load time, 1 minute the mindless task, 1 minute reorganizing stuff to have it perform better. There is clearly room for improvement because now the 'robot' logs in every 5 minutes, does the job and gets out. Initial estimation shows it earns about 80% more virtual money than I was able (wanting) to.
Organization
The playing area is composed of several rectangles that you have to swipe through. Above you see only one are but my current field is about 5x5 this.
- The small houses with the gold floating on top of them should be clicked every 5 minutes or so.
- The animal in the top right part should be clicked evey 2 hours or so.
- The green farm bottom center should be activated every 10 hours or so by clicking on the Activate button a few times and then on the farm itself.
Reverse-engineering
The app is a crappy port, probably from iOS, so most if its part is native. It also uses a custom game engine that I haven't been able to find any info on. The game logic itself is stored in some binary files that I assume to be some recompiled Lua scripts.Seeing that this would take too long I went down the brute-force approach. It would also work if the app were to be updated, which in this case happens at around two weeks. That's right, they are pushing 80-150MB every two weeks to every device in the user base!
Idea
I envisioned the initial implementation to just follow my moves. It is close to impossible to find a macro recorder for Android so I rolled my own.
Since the app was native it was not possible to click by finding out objects inside the Window Manager, it must be done via the low-level [hardware] event manager.
First, listing input devices:
root@android:/ # getevent -p
getevent -p
add device 1: /dev/input/event6
name: "HPTouchpad"
events:
ABS (0003): 0030 : value 0, min 0, max 0, fuzz 0, flat 0, resolution 0
0035 : value 0, min 0, max 1024, fuzz 2, flat 0, resolution 0
0036 : value 0, min 0, max 768, fuzz 1, flat 0, resolution 0
0039 : value 0, min 0, max 0, fuzz 0, flat 0, resolution 0
input props:
<none>
add device 2: /dev/input/event1
name: "pmic8058_pwrkey"
events:
KEY (0001): 006b 0074
input props:
<none>
add device 3: /dev/input/event2
name: "lsm303dlh_mag_sysfs"
events:
ABS (0003): 0000 : value -434, min -8100, max 8100, fuzz 0, flat 0, resolution 0
0001 : value 848, min -8100, max 8100, fuzz 0, flat 0, resolution 0
0002 : value -950, min -8100, max 8100, fuzz 0, flat 0, resolution 0
input props:
<none>
add device 4: /dev/input/event3
name: "lsm303dlh_acc_sysfs"
events:
ABS (0003): 0000 : value -19, min -8000, max 8000, fuzz 0, flat 0, resolution 0
0001 : value -12, min -8000, max 8000, fuzz 0, flat 0, resolution 0
0002 : value 1029, min -8000, max 8000, fuzz 0, flat 0, resolution 0
0008 : value 0, min -2147483648, max 2147483647, fuzz 0, flat 0, resolution 0
0028 : value 0, min -2147483648, max 2147483647, fuzz 0, flat 0, resolution 0
input props:
<none>
add device 5: /dev/input/event4
name: "isl29023 light sensor"
events:
ABS (0003): 0028 : value 12, min 0, max 798400, fuzz 0, flat 0, resolution 0
input props:
<none>
add device 6: /dev/input/event5
name: "headset"
events:
KEY (0001): 00a4
SW (0005): 0002 0004
input props:
<none>
add device 7: /dev/input/event0
name: "gpio-keys"
events:
KEY (0001): 0072 0073 00e8 00f9
input props:
<none>
Just by name we can figure out that event6 events are pushed by the touchscreen, event1 is the power key, event2 is the magnetometer, event3 the accelerometer, event4 the light sensor, event5 the headphone jack detection and event0 the remaining hardware keys (volume buttons, AFAIK).
Recording the events
it's just a matter of running getevent either on the device itself or via adb. Something like "getevent >events.out". The output should be similar to:
/dev/inputevent3: 0003 0000 fffffff1
/dev/inputevent3: 0003 0001 ffffffeb
/dev/inputevent3: 0003 0002 000003fd
/dev/inputevent3: 0000 0000 00000000
/dev/inputevent3: 0003 0000 fffffffb
/dev/inputevent3: 0003 0001 fffffff2
/dev/inputevent3: 0003 0002 000003fd
/dev/inputevent3: 0000 0000 00000000
/dev/inputevent6: 0003 0030 00000032/dev/inputevent6: 0003 0035 00000217/dev/inputevent6: 0003 0036 000000b8/dev/inputevent6: 0000 0002 00000000/dev/inputevent6: 0000 0000 00000000/dev/inputevent3: 0003 0000 fffffff9/dev/inputevent3: 0003 0001 fffffff0/dev/inputevent3: 0003 0002 000003f8/dev/inputevent3: 0000 0000 00000000/dev/inputevent6: 0003 0039 00001988/dev/inputevent6: 0003 0030 00000032/dev/inputevent6: 0003 0035 00000217/dev/inputevent6: 0003 0036 000000b8/dev/inputevent6: 0000 0002 00000000/dev/inputevent6: 0000 0000 00000000/dev/inputevent6: 0003 0039 00001988
You might spot a problem here that I did not recognize in the beginning. Anyway, onward to
Playing back the events:
During some kind of twisted design decision it was decided that the "sendevent" command has a different format than what "getevent" provides. Luckily I found a blog post that had an almost working solution written in Python that did everything:
The problems
Everything went nice but for a recorded session lasting about 30 seconds the script took about 15 minutes to replay.
Also, all the micro-swipes were being converted to clicks and getting lost and all the swipe inertia was being lost because the speed was much lower. You normally don't realize all this stuff happening behind the scenes this when using a capacitive touchscreen.
The first issue could be dealt with by doing a few optimizations. You can see from the listing above that for every touch event there is are a ton of accelerometer events. A simple grep takes care of that.
Also, sending events from the PC to the phone is really slow so it can be solved like this:
:start@time /tcall monkeyrunner.bat %CD%\mainscreen.pyadb push mainscreen.scr /sdcard/mainscreen.scradb shell sh /sdcard/mainscreen.scr@time /ttimeout /t 30goto startpause
Where the python script just unlocks the screen, kills the app if it is running and starts it again. The mainscreen.scr file contains the sendevent commands line by line:
#!/bin/sh
echo Running - drawing function
sendevent /dev/input/event6 3 53 663
sendevent /dev/input/event6 3 54 116
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 663
sendevent /dev/input/event6 3 54 116
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 663
sendevent /dev/input/event6 3 54 116
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 657
sendevent /dev/input/event6 3 54 127
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 652
sendevent /dev/input/event6 3 54 138
sendevent /dev/input/event6 0 2 0
sendevent /dev/input/event6 0 0 0
sendevent /dev/input/event6 3 53 645
Not all events are needed so some of them were stripped, I forgot which, but probably touch area and several of the touch confirmations. Here's a draft I had in the folder, ending with a parsed getevent.
Multi-touch devices use the following Linux input events:
ABS_MT_POSITION_X: (REQUIRED) Reports the X coordinate of the tool.
ABS_MT_POSITION_Y: (REQUIRED) Reports the Y coordinate of the tool.
ABS_MT_PRESSURE: (optional) Reports the physical pressure applied to the tip of the tool or the signal strength of the touch contact.
ABS_MT_TOUCH_MAJOR: (optional) Reports the cross-sectional area of the touch contact, or the length of the longer dimension of the touch contact.
ABS_MT_TOUCH_MINOR: (optional) Reports the length of the shorter dimension of the touch contact. This axis should not be used if ABS_MT_TOUCH_MAJOR is reporting an area measurement.
ABS_MT_WIDTH_MAJOR: (optional) Reports the cross-sectional area of the tool itself, or the length of the longer dimension of the tool itself. This axis should not be used if the dimensions of the tool itself are unknown.
ABS_MT_WIDTH_MINOR: (optional) Reports the length of the shorter dimension of the tool itself. This axis should not be used if ABS_MT_WIDTH_MAJOR is reporting an area measurement or if the dimensions of the tool itself are unknown.
ABS_MT_ORIENTATION: (optional) Reports the orientation of the tool.
ABS_MT_DISTANCE: (optional) Reports the distance of the tool from the surface of the touch device.
ABS_MT_TOOL_TYPE: (optional) Reports the tool type as MT_TOOL_FINGER or MT_TOOL_PEN.
ABS_MT_TRACKING_ID: (optional) Reports the tracking id of the tool. The tracking id is an arbitrary non-negative integer that is used to identify and track each tool independently when multiple tools are active. For example, when multiple fingers are touching the device, each finger should be assigned a distinct tracking id that is used as long as the finger remains in contact. Tracking ids may be reused when their associated tools move out of range.
ABS_MT_SLOT: (optional) Reports the slot id of the tool, when using the Linux multi-touch protocol 'B'. Refer to the Linux multi-touch protocol documentation for more details.
BTN_TOUCH: (REQUIRED) Indicates whether the tool is touching the device.
BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_BACK, BTN_SIDE, BTN_FORWARD, BTN_EXTRA, BTN_STYLUS, BTN_STYLUS2: (optional) Reports button states.
BTN_TOOL_FINGER, BTN_TOOL_PEN, BTN_TOOL_RUBBER, BTN_TOOL_BRUSH, BTN_TOOL_PENCIL, BTN_TOOL_AIRBRUSH, BTN_TOOL_MOUSE, BTN_TOOL_LENS, BTN_TOOL_DOUBLETAP, BTN_TOOL_TRIPLETAP, BTN_TOOL_QUADTAP: (optional) Reports the tool type.
[ 71418.045740] EV_ABS ABS_MT_TRACKING_ID 0000215d[ 71418.045789] EV_ABS ABS_MT_TOUCH_MAJOR 00000019[ 71418.045813] EV_ABS ABS_MT_POSITION_X 0000026a[ 71418.045837] EV_ABS ABS_MT_POSITION_Y 0000017a[ 71418.045861] EV_SYN SYN_MT_REPORT 00000000[ 71418.045882] EV_SYN SYN_REPORT 00000000[ 71418.056783] EV_ABS ABS_MT_TRACKING_ID 0000215d[ 71418.056838] EV_ABS ABS_MT_TOUCH_MAJOR 00000019[ 71418.056862] EV_ABS ABS_MT_POSITION_X 0000026a[ 71418.056885] EV_ABS ABS_MT_POSITION_Y 0000017a[ 71418.056908] EV_SYN SYN_MT_REPORT 00000000[ 71418.056929] EV_SYN SYN_REPORT 00000000[ 71418.067805] EV_ABS ABS_MT_TRACKING_ID 0000215d
All these changes improved the time from 15 minutes down to 5 minutes, but it was not good enough for 'production'.
But the most important problem of this approach was that it had no provision for application layout update and required a new recording with each game field change (i.e. moving animals around). Recording including parsing and everything took around 2-5 minutes per session and was probably required every two days.
Comments
Post a Comment
Due to spammers, comments sometimes will go into a moderation queue. Apologies to real users.