Adventure with Android. Part 5. LUA.
.APK file does not contain all necessary files and files I was looking for were stored here:
/data/data/<game packageid>/files/<version of app>
But that time I didn’t know that and as result I came to conclusion that this thing is done from code. And I decided to decompile game logic. This particular game uses LUA language for most things. This is quite simple language, but it is compiled to byte code and needs to be decompiled back to source code. I immediately found this https://github.com/viruscamp/luadec and I did attempt to decompile simple file but it instantly failed. After some research and looking at simplest file I found that it has only one instruction: Return and it has code 0x1A. Header of that file stated that it is LUA version 5.1. And according to documentation for version 5.1 return instruction should has code 0x1E.
After looking on history of LUA I found that new codes were added few times before return instruction and as result I decided that header is invalid, and I needed older version of LUA. After spending few hours on searching I found really old beta version of LUA 5.0 that has that return instruction with code 0x1A. Only to find later that version is really old. I don’t remember exactly but something like 2006. Later I found in .lua files that it is compiled with relatively new version of cocoa (you find string cocos2d-x-2.2.6), I came to conclusion that it is dead end and header actually has correct value and it is LUA 5.1. Somebody just messed up byte codes to make it harder to decompile.
And at that moment it became personal. I started to think how I can convert game byte codes to correct LUA 5.1 byte code. And after some thoughts I decided to do similar to decryption. I decided that game should compile my source .lua file and produce compiled binary .lua with encoded byte codes. Then I will compare compiled binary version. And at the end I wrote something like this:
typedef lua_State* (*lua_openFunc)();
typedef void (*lua_closeFunc)(lua_State* L);
typedef int (*luaL_loadfileFunc)(lua_State* L, const char* filename);
typedef int (*lua_pcallFunc)(lua_State* L, int nargs, int nresults, int msgh);
typedef void (*luaopen_baseFunc)(lua_State* L);
typedef void (*luaopen_tableFunc)(lua_State* L);
typedef void (*luaopen_ioFunc)(lua_State* L);
typedef void (*luaopen_stringFunc)(lua_State* L);
typedef void (*luaopen_mathFunc)(lua_State* L);
typedef void (*luaL_openlibsFunc)(lua_State* L);
typedef const char* (*lua_tostringFunc)(lua_State* L, int index, size_t* len);
lua_openFunc lua_open;
lua_closeFunc lua_close;
luaL_loadfileFunc luaL_loadfile;
lua_pcallFunc lua_pcall;
luaopen_baseFunc luaopen_base;
luaopen_tableFunc luaopen_table;
luaopen_ioFunc luaopen_io;
luaopen_stringFunc luaopen_string;
luaopen_mathFunc luaopen_math;
luaL_openlibsFunc luaL_openlibs;
lua_tostringFunc lua_tostring;
void SetupLuaExports(void* handle)
{
lua_open = (lua_openFunc)dlsym(handle, "luaL_newstate");
if (lua_open == nullptr)
{
LOGW("DH: Failed to resolve function");
return;
}
lua_close = (lua_closeFunc)dlsym(handle, "lua_close");
if (lua_close == nullptr)
{
LOGW("DH: Failed to resolve function");
return;
}
luaL_loadfile = (luaL_loadfileFunc)dlsym(handle, "luaL_loadfile");
if (luaL_loadfile == nullptr)
{
LOGW("DH: Failed to resolve function");
return;
}
lua_pcall = (lua_pcallFunc)dlsym(handle, "lua_pcall");
if (lua_pcall == nullptr)
{
LOGW("DH: Failed to resolve function");
return;
}
luaL_openlibs = (luaL_openlibsFunc)dlsym(handle, "luaL_openlibs");
if (luaL_openlibs == nullptr)
{
LOGW("DH: Failed to resolve function");
return;
}
lua_tostring = (lua_tostringFunc)dlsym(handle, "lua_tolstring");
if (lua_tostring == nullptr)
{
LOGW("DH: Failed to resolve function");
return;
}
}
...
SetupLuaExports(handle);
load("/storage/emulated/0/Android/data/com.Android2/files/main.lua");
Then inspired by luac.lua file from LUA 5.1 installation I put following file:
-- bare-bones luac in Lua
-- usage: lua luac.lua file.lua
f=assert(io.open("/storage/emulated/0/Android/data/com.Android2/files/luac.out","wb"))
assert(f:write(string.dump(assert(loadfile("/storage/emulated/0/Android/data/com.Android2/files/main.lua")))))
assert(f:close())
Then I run my application and it successfully generated luac.out file that contains encoded byte code. I already know that 0x1A should be mapped to 0x1E and after some testing, I found two other pairs. But it was manual process and it is quite error prone. And the end I decided that computer should do mapping for me. And for this I need source .lua file that will generate all possible byte codes. I found nice document ANoFrillsIntroToLua51VMInstructions.pdf and after about hour I create this .lua file:
function z12()
end
function z13()
return z12()
end
local a,b = 10; b = a
local c,d,e = nil,nil,0
local f,g = false, true
local a4 = 5 > 2; a5 = 5 >= 2; a6 = 5 < 2; a7 = 5 <= 2; a7 = 5 == 2; a8 = 5 ~= 2
a5 = 40; local b5 = a5
local a6; function b6() a6 = 1 return a6 end
local a7 = {}; a7[1] = "foo"; local b7 = a7["bar"]
local a8,b8 = 2,4; a8 = a8 + 4 * b8 - a8 / 2 ^ b8 % 3
local p9,q9 = 10,false; q9,p9 = -p9,not q9
local a10,b10; a10 = #b10; a10= #"foo"
local x11,y11 = "foo","bar"; a11 = x11..y11..x11..y11
z12()
z13()
local a14,b14,c14 = ...
foo:bar("baz")
local a16,b16,c16; c16 = a16 and b16
local a17,b17; a17 = a17 or b17
local a18 = 0; for i18 = 1,100,5 do a18 = a18 + i18 end
for i19,v19 in pairs(t1) do print(i19,v19) end
local q20 = {}
local q21 = {1,2,3,4,5,}
do
local p22,q22
r = function() return p22,q22 end
end
Note: After I finished decompiling, I was told that that decompiler already contains such file: luadec\bin\allopcodes-5.1.lua
Anyway, after that step I have encoded binary lua and after I compiled the same file using regular LUA 5.1, I have normal binary lua file for the same source file. I couldn’t compile it exactly the same and closest one always has source file name inside of binary lua file. And I have to write this small app that removes this file name:
class Program
{
static void Main(string[] args)
{
StripInfo(args[0], args[1]);
}
private static void StripInfo(string inputFile, string outputFile)
{
byte[] header = new byte[12];
using (var inputStream = File.OpenRead(inputFile))
{
using (var outputStream = File.Create(outputFile))
{
if (inputStream.Read(header, 0, header.Length) != header.Length)
{
throw new InvalidOperationException("Failed to read header");
}
outputStream.Write(header, 0, header.Length);
if (inputStream.Read(header, 0, Marshal.SizeOf<int>()) != Marshal.SizeOf<int>())
{
throw new InvalidOperationException("Failed to read length");
}
int skipBytes = BitConverter.ToInt32(header, 0);
for(int i = 0; i< Marshal.SizeOf<int>();i++)
{
header[i] = 0;
}
outputStream.Write(header, 0, Marshal.SizeOf<int>());
inputStream.Seek(skipBytes, SeekOrigin.Current);
inputStream.CopyTo(outputStream);
}
}
}
}
After that both files have exactly the same length. After comparing files, I found that only bytes codes are changed and arguments for instructions are exactly the same. And as result I have to write another app that build decoding and encoding tables:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;
namespace BuildTransform
{
internal class TransformBuilder
{
public TransformBuilder()
{
}
internal void Run(string inputFile1, string inputFile2)
{
using (var inputStream1 = File.OpenRead(inputFile1))
{
using (var inputStream2 = File.OpenRead(inputFile2))
{
inputStream1.Seek(0xC, SeekOrigin.Begin);
inputStream2.Seek(0xC, SeekOrigin.Begin);
Dictionary<int, int> processedInstructions = new Dictionary<int, int>();
ProcessInstructions(inputStream1, inputStream2, processedInstructions);
foreach (int inst1 in processedInstructions.Keys.OrderBy(i => i))
{
int inst2 = processedInstructions[inst1];
if (inst1 == inst2)
{
Console.WriteLine($"G: {inst1:X2} -> {inst2:X2}");
}
else
{
Console.WriteLine($"B: {inst1:X2} -> {inst2:X2}");
}
}
}
}
}
private void SkipConstants(Stream stream)
{
int count = ReadInt(stream);
for(int i = 0; i< count;i++)
{
SkipConstant(stream);
}
}
private void SkipConstant(Stream stream)
{
int type = ReadByte(stream);
switch(type)
{
case 3:
SkipIntegerConst(stream);
break;
case 4:
SkipStringConst(stream);
break;
default:
throw new InvalidOperationException($"Invalid constant type {type}");
}
}
private void SkipIntegerConst(Stream stream)
{
ReadInt(stream); // Skip value
ReadInt(stream); // Skip value
}
private void SkipStringConst(Stream stream)
{
int length = ReadInt(stream);
stream.Seek(length, SeekOrigin.Current);
}
private void ProcessInstructions(FileStream inputStream1, FileStream inputStream2, Dictionary<int, int> processedInstructions)
{
SkipHeader(inputStream1);
SkipHeader(inputStream2);
int ic1 = ReadInt(inputStream1);
int ic2 = ReadInt(inputStream2);
if (ic1 != ic2)
{
throw new InvalidOperationException($"File 1 has {ic1} intructions but file 2 has {ic2} intructions");
}
for (int i = 0; i < ic1; i++)
{
int inst1 = ReadInstruction(inputStream1);
int inst2 = ReadInstruction(inputStream2);
if (!processedInstructions.ContainsKey(inst1))
{
processedInstructions.Add(inst1, inst2);
}
}
SkipConstants(inputStream1);
SkipConstants(inputStream2);
int fc1 = ReadInt(inputStream1);
int fc2 = ReadInt(inputStream2);
if (fc1 != fc2)
{
throw new InvalidOperationException($"File 1 has {fc1} functions but file 2 has {fc2} functions");
}
for (int i = 0; i < fc1; i++)
{
ProcessInstructions(inputStream1, inputStream2, processedInstructions);
}
SkipLines(inputStream1);
SkipLines(inputStream2);
SkipLocVars(inputStream1);
SkipLocVars(inputStream2);
SkipUpValues(inputStream1);
SkipUpValues(inputStream2);
}
private void SkipLines(Stream stream)
{
int count = ReadInt(stream);
for(int i = 0; i< count;i++)
{
ReadInt(stream);
}
}
private void SkipLocVars(Stream stream)
{
int count = ReadInt(stream);
for (int i = 0; i < count; i++)
{
SkipLocVar(stream);
}
}
private void SkipLocVar(Stream stream)
{
int length = ReadInt(stream);
stream.Seek(length, SeekOrigin.Current);
ReadInt(stream); // Start PC
ReadInt(stream); // End PC
}
private void SkipUpValues(Stream stream)
{
int count = ReadInt(stream);
for(int i = 0; i< count;i++)
{
SkipUpValue(stream);
}
}
private void SkipUpValue(Stream stream)
{
int length = ReadInt(stream);
stream.Seek(length, SeekOrigin.Current);
}
private void SkipHeader(Stream stream)
{
stream.Seek(0x10, SeekOrigin.Current);
}
private int ReadInstruction(Stream stream)
{
int result = ReadInt(stream);
return result & 0x3F;
}
private byte ReadByte(Stream stream)
{
int result = stream.ReadByte();
if(result == -1)
{
throw new InvalidOperationException("Failed to read from file");
}
return (byte)result;
}
private int ReadInt(Stream stream)
{
byte[] data = new byte[4];
if (stream.Read(data, 0, 4) != 4)
{
throw new InvalidOperationException("Failed to read from file");
}
return BitConverter.ToInt32(data, 0);
}
}
}
And you use it like this:
new TransformBuilder().Run(args[0], args[1]);
And application will output decoding values and with slight modification also encoding tables. First thing I did to change GET_OPCODE macros in lopcodes.h to this:
(cast(OpCode, DoDecodeTransform(((i)>>POS_OP) & MASK1(SIZE_OP,0))))
And then I added this to lopcodes.c
const char DecodeTransform[NUM_OPCODES] =
{
0x0D,
0x05,
0x1A,
0x06,
0x01,
0x03,
0x1D,
0x16,
0x21,
0x13,
0x14,
0x23,
0x19,
0x24,
0x09,
0x1B,
0x10,
0x04,
0x20,
0x0E,
0x0A,
0x0B,
0x15,
0x02,
0x22,
0x12,
0x1E,
0x0F,
0x00,
0x07,
0x0C,
0x17,
0x1F,
0x18,
0x11,
0x08,
0x1C,
0x25
};
char DoDecodeTransform(char value)
{
return DecodeTransform[value];
}
But when I attempt to compile whole solution it failed because decompiler actually includes full source code of LUA 5.1 and after it compiles it, there is step that builds some table and it includes running LUA file. Obviously it will fail because it expects normal commands and not encoded one. As result I had to add flag and by default it set to zero and luadec.c in main function I set it to 1. And final code looks like that:
// lopcodes.c
char DoDecode = 0;
char DoDecodeTransform(char value)
{
if (DoDecode)
{
return DecodeTransform[value];
}
else
{
return value;
}
}
// lopcodes.h
LUAI_DATA char DoDecode;
LUAI_DATA char DoDecodeTransform(char value);
//luadec.c
int main(int argc, char* argv[]) {
DoDecode = 1;
...
Then I wrote small batch file:
for /R %%I in (*.lua) do C:\Projects\IdleHeros.Data\Lua\luadec\vcproj-5.1\Debug\bin\luadec.exe %%I > %%I.src
for /R %%I in (*.lua) do C:\Projects\IdleHeros.Data\Lua\luadec\vcproj-5.1\Debug\bin\luadec.exe -dis %%I > %%I.dis
First line decompiles to more or less readable source file and second one disassembles and outputs instructions with arguments. Decompiling even such simple language is not easy thing and, in many cases, decompiler will fail. So, as backup I added disassembling step. It will always succeed. It is hard to read but still possible and it will provide help when decompiler will fail.
But then I have idea. In order to get some data from .lua files I have to find it in different files. I’m lazy and I think it will be better to write lua program that will load binary .lua files, combine data from them and output result in format I prefer. I checked this idea and it was working just fine. But I have to constantly copy files to and from emulator. And it takes a lot of time to run application etc. And then I have another idea. What if I will run lua program on my PC? I can change compiler to encode bytes codes, so during source code compilation it will output the same values as encoded version from game. And then I can just run it because it already has part that will decode byte codes. And as result my program will be compiled exactly the same way as game itself.
And to do this I did following modifications:
// lopcodes.h
#define SET_OPCODE(i,o) ((i) = DoEncodeTransform((((i)&MASK0(SIZE_OP,POS_OP)) | \
((cast(Instruction, o)<<POS_OP)&MASK1(SIZE_OP,POS_OP)))))
...
#define CREATE_ABC(o,a,b,c) ((cast(Instruction, DoEncodeTransform(o))<<POS_OP) \
| (cast(Instruction, a)<<POS_A) \
| (cast(Instruction, b)<<POS_B) \
| (cast(Instruction, c)<<POS_C))
#define CREATE_ABx(o,a,bc) ((cast(Instruction, DoEncodeTransform(o))<<POS_OP) \
| (cast(Instruction, a)<<POS_A) \
| (cast(Instruction, bc)<<POS_Bx))
...
LUAI_DATA char DoDecode;
LUAI_DATA char DoEncode;
LUAI_DATA char DoDecodeTransform(char value);
LUAI_DATA char DoEncodeTransform(char value);
//lopcodes.c
char DoDecode = 1;
char DoEncode = 1;
char DoDecodeTransform(char value)
{
if (DoDecode)
{
return DecodeTransform[value];
}
else
{
return value;
}
}
char DoEncodeTransform(char value)
{
if (DoEncode)
{
return EncodeTransform[value];
}
else
{
return value;
}
}
And after that I was able to load any binary .lua files I want, use data from them etc. Keep in mind that if you would like to recompile decompiler then you have to change default value for DoDecode to zero or it will fail.
And here my journey finishes. It will quite fun and I learnt a lot from it. I hope it will be helpful to someone.