Mittwoch, 20. April 2011

Deobfuscating java projects

Lately I've been digging into minecraft. Normally digging into minecraft means something like this:

I was rather digging into something like this:

/*    */ import java.io.DataInputStream;
/*    */ import java.io.DataOutputStream;
/*    */ 
/*    */ public class a extends gc
/*    */ {
/*    */   public int a;
/*    */   public int b;
/*    */   public int c;
/*    */ 
/*    */   public void a(DataInputStream paramDataInputStream)
/*    */   {
/* 21 */     this.a = paramDataInputStream.readInt();
/* 22 */     this.b = paramDataInputStream.readInt();
/* 23 */     this.c = paramDataInputStream.readByte();
/*    */   }
/*    */ 
/*    */   public void a(DataOutputStream paramDataOutputStream) {
/* 27 */     paramDataOutputStream.writeInt(this.a);
/* 28 */     paramDataOutputStream.writeInt(this.b);
/* 29 */     paramDataOutputStream.writeByte(this.c);
/*    */   }

Obfuscated java code :-(

What is obfuscated code?

By obfuscating your java classes you want to prevent others from taking your source, modify it and release it again under another brand. There are tools which automatically obfuscate your code. These tools normally do the following things:

  • Put all artifacts (classes, interfaces, ...) into the default package
  • Rename all artifacts and members to something generic like 'a'

These changes make it very hard for humans to understand the meaning of some code. But there are a few things that can't be changed that easily during obfuscation:

  • Access modifiers
  • Types
  • Signatures in general
  • Import statements

What has all this to do with minecraft?

mojang is obfuscating it's minecraft releases so nobody can steal their code. Unfortunately this makes it very hard for developers to extend minecraft due to the lack of APIs within minecraft. So the community helped itself and created bukkit. bukkit is an API for minecraft. bukkit is so important just because of the minecraft obfuscation. With every minecraft release (major or minor) the signatures of all classes change. bukkit is the abstraction layer for the obfuscation changes.

But this means a lot of work for the bukkit developers [1]. They need to revert the minecraft obfuscation for every minecraft release manually.

The plan

Now there is a chance to make undoing the obfuscation more easily. But it will only work when the following conditions are met:

  • You got the deobfuscated source of a former version
  • The software is implemented iterative. So only relative small amounts of the code change during two versions.

With these assumptions you can statistically try to match artifacts from the old source with artifacts from the new obfuscated source. You just need to compare the things that don't change during obfuscation. Theres a big chance that the two artifacts with the most equality are actually the same artifact. This allows you to transfer the artifact's name and package from the old source into the new source.

The project

I like the idea of automatically deobfuscate java code. So I've hacked something and released it on github. java-deobscurify tries to find matching artifacts in the clear text and obfuscated source. java-deobscurify is still a work in progress. Let's say pre alpha... but I hope it will produce reasonable output in the near future.

References

  1. When will Bukkit be updated for Minecraft 1.5?

Mittwoch, 30. März 2011

Another minecraft rendering

Another minecraft povray rendering got finished. For more details see my former blog post Rendering minecraft worlds with povray:

Montag, 21. März 2011

Rendering minecraft worlds with povray

Lately I've been playing with the minecraft savegames. There's not much documentation about the savegame format [1] [2]. But I've managed to export some blocks and render them with povray. Here are the results:

References

  1. RegionFileCache.java
  2. RegionFile.java

Samstag, 19. Februar 2011

Headless HTML page rendering with phantomjs

What is phantomjs?

phantomjs is a headless browser which can render HTML pages into images. It uses a Webkit rendering engine. That's cool because the generated page images aren't missing any dynamically loaded javascript stuff. Event flash movies are shown as expected.

Building phantomjs

The phantomjs homepage contains some very useful hints about getting and building phantomjs. Read the comments at the end of the build instructions page if something doesn't work out for you. There are many useful hints.

Rendering my first page

phantomjs is controlled using javascript commands. You can launch your phantomjs javascript files from the commandline:

$ phantomjs myScript.js

phantomjs calls your script every time after a page is loaded. To react in different ways on different pages you have to track your current "state". You can persist your script's state in the var phantom.state. phantomjs restores the value of phantom.state even after another page is loaded. Other javascript variables disappear after loading a new page. Initially the state is empty.

To load a page call phantom.open(url). phantomjs will load the page below the specified URL and execute your script after the page's DOM is loaded.

The (in my opinion) coolest function is phantom.render(path). phantomjs will save a "screenshot" of your current page to the specified path on your hard drive. The path's file extension defines the image file format. Your viewport's size can be defined by setting phantom.viewportSize.

After you're done with your phantom things you have to call phantom.exit(returnCode) to quit. The returnCode is passed as status to the parent process.

Now putting all that functions together in a script gives us the following:

if(phantom.state.length === 0){
  phantom.state = '0_home';
  phantom.open('http://www.mini.de');
}
else if(phantom.state === '0_home'){
  phantom.viewportSize = {width: 800, height: 600};
  phantom.sleep(2000);
  phantom.render('home.png');
  phantom.exit(0);
}

The result after executing the script is a screenshot of www.mini.de in a file named 'home.png'. The image will show the main stage with a flash movie and a footer with three dynamically loaded HTML fragments. You can see the result below:

Clicking links

"Clicking links" like a humanoid user is simulated by firing mouse click events. The following listing defines the clickElement(id) function which can be used to "click" elements. This allows us to execute simple page flows.

function clickElement(id){
  var a = document.getElementById(id);

  var e = document.createEvent('MouseEvents');
  e.initMouseEvent('click', true, true, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null);

  a.dispatchEvent(e);
}

if(phantom.state.length !== 0){
  // save screenshot for every page / state
  phantom.viewportSize = {width: 800, height: 600};
  phantom.sleep(2000);
  phantom.render('screen_' + phantom.state + '.png');
}

if(phantom.state.length === 0){
  phantom.state = '0_home';
  phantom.open('http://www.mini.de');
}
else if(phantom.state === '0_home'){
  phantom.state = '1_config';

  clickElement('quicklink_id1');
}
else if(phantom.state === '1_config'){
  phantom.exit();
}

The script will load the URL http://www.mini.de which is the home page. After loading the home page the element with the ID 'quicklink_id1' will be target of a click event. The element with the ID 'quicklink_id1' should be the 'MINI KONFIGURATOR' link in the footer.

Asserting stuff

Within phantomjs scripts you can access you page's DOM, the global javascript variables and global javascript functions. This enables us to do some kind of unit testing.

I'm extending the listing once more:

function clickElement(id){ ... }

function fail(msg){
 console.log(msg);

 phantom.exit(1);
}

function assert(condition, msg){
 if(condition){
  return;
 }

 fail(msg);
}

if(phantom.state.length !== 0){
 phantom.viewportSize = {width: 800, height: 600};
 phantom.sleep(2000);
 phantom.render('screen_' + phantom.state + '.png');
}

if(phantom.state.length === 0){
 phantom.state = '0_home';
 phantom.open('http://www.mini.de');
}
else if(phantom.state === '0_home'){
 phantom.state = '1_config';

 clickElement('quicklink_id1');
}
else if(phantom.state === '1_config'){
 assert(document.getElementById('cake'), 'I am missing the cake!');

 phantom.exit();
}

Check this out: phantomjs returns a status code of 1 when the assertion fails. This can be used to do some custom post processing in the shell:

$ phantomjs myScript.js && echo 'Test successful' || echo 'Test failed'

Further information

For further information see the following pages. They are sorted by what I think is important.

Freitag, 3. September 2010

experimental freebase integration in tagfs

Today I made a little experiment using tagfs and freebase. Freebase is a big database with structured data which contains around 12 million entities [1]. These entities are used to describe subjects from the real world. Like movies, persons, places, ... So freebase is perfectly applicable for enriching your own data.

I will show this by an example. I've got some movies on my hard drive which I enjoy watching. One of them is the movie Ink. I store these movies in a movie directory. Every movie gets it's own subdirectory below the movies directory. So the movie Ink is stored on my hard drive below /movies/Ink/Ink.avi. This enables me to tag my movies for tagfs. My tag file for Ink looks like this:

My Personal Rating: 8 stars
_freebase: type: /film/film, name: Ink, initial_release_date: None

In my experiment I extended tagfs to fetch taggings from freebase. tagfs reads the _freebase value for each item / movie and executes it as a query to freebase. The query delivers a result like this:

{
  "code": "/api/status/ok",
  "result": {
    "initial_release_date": [
      "2009-11-10"
    ],
    "name": "Ink",
    "type": "/film/film"
  },
  "status": "200 OK",
  "transaction_id": "cache;cache03.p01.sjc1:8101;2010-09-03T12:38:45Z;0028"
}

tagfs applies the fields initial_release_date, name and type as further taggings. So exporting the tagfs meta data as CSV looks like this:

"name";"name";"type";"initial_release_date";"Titel"
"Ink";"Ink";"_film_film";"2009-11-10";"Ink"

Now I can filter my movies with tagfs using the meta data from the huge freebase community. I've release the experimental freebase integration in tagfs below the freebase branch.

References

Montag, 16. August 2010

using a file system for my bank account

the goal - organize your shared payments

I'm sharing a flat with my girl friend. So we regularly shop things, that we are equally paying. Sometimes it's a little hard to track the many shared payments. Especially if you use cash, your giro account and visa.

the idea - use a directory

To track all your payments you need to organize all payments in one place. Normally this can't be your giro account as your bank doesen't care about your cash payments. So my approach is to gather my payment data on my computer. Fortunately my bank supports a CVS export of my giro and visa transactions. So I can easily export the account data in a computer processable format.

the database is a directory

As database for my payments I use a plain old directory on my file system. The directory contains a subdirectory for every transaction I made. No matter whether it was on my bank accounts or cash. For the transaction directory names I use the simple syntax: <date> - <description>

Examples for the transaction directory names would be:

  • 2010-08-12 - ice cream in park
  • 2010-08-14 - GIRO PAYMENT at MY GROCERIES SHOP
  • 2010-08-14 - VISA PAYMENT at furniture.example.com

Each of the transaction directories is tagged with the transaction details. The taggings are applied in a text file within the transaction directory. The tagging file 2010-08-14 - GIRO PAYMENT at MY GROCERIES SHOP/.tag i.e. has the following content:

date: 2010-08-14
account: visa
amount EUR: 120.00
description: old table

The tag files for the giro and visa transactions are created by a python script I wrote. The python script creates a transaction directory for every row in my bank's CSV exports. The columns from the CSV files are applied as taggings in the tag file. The python script does a little more magic, like merging already existing entries. But that's another story.
The transaction directories for cash payments are created manually by me. But the taggings contain basically the same data. The only difference is that the account field is tagged with cash.

I introduce a new tagging to separate my payments from the payments I share with my girl friend. My tagging for shared payments is:

share: true

export filtered CSV with tagfs

Now that I've collected my transactions and added meta data like the 'share' tagging, I need to filter the transactions. I use tagfs to filter the various transactions, contexts (like 'data', 'account', ...) and taggings. tagfs is mounted as a virtual file system beside the transactions directory. Mounting the above example with tagfs will show me a directory like this:

The tagfs root directory contains various subdirectories. The subdirectories represent filters for the tagfs items aka. my transaction directories. Filtering transactions takes place by entering directories. Enter the share/true/ directory to see all transactions with a share flag. share/true/account/giro will show you all shared transactions which occurred via your giro account.

Now I create a CSV export which contains all shared transactions. To do so I open the CSV file share/true/.export/export.csv. The CSV file contains all matching transactions as rows. The columns represent the different taggings:

name date account amount EUR description share
2010-08-12 - ice cream in park 2010-08-12 cash 1.00   true
2010-08-14 - GIRO PAYMENT at MY GROCERIES SHOP 2010-08-14 giro 32.53 just food true

I open this CSV table in OpenOffice.org and calculate the sum in the 'amount EUR' column. That's real magic... isn't it?!?

adding multi dimensional spice

OK... I truly admit... the magic hasen't happend yet. What I just did was just some filter with a sum calculation. I could have used a simple excel sheet for that. Excel is a fine tool as long as you use structured data. Our transactions are structured data. Every transactions consists of a limited amount of fields with well defined values. To leave this limited view of the world you have to think of subjects instead of transactions in your bank account. A subject can be anything! A transactions can be a subject as good as a directory with my holiday pictures can be a subject (I borrowed this very abstract view of subjects from the resource description framework and the tripplestore concept). Now I can tag my transactions and my holiday pictures with holiday: india 2009. This allows me various filters:

  • holiday/india 2009 shows everything related to my india vacation in 2009. No matter whether it's a transaction on my visa account or my holiday pictures.
  • holiday/india 2009/account/visa/.export/export.csv lets me calculate all my visa expenses during the holiday.

(my) conclusion

As I think, an excel sheet or a relational database system gives you one view to your data. Viewing your data in a table like structure is good for analyzing items. These items need to be comparable in a specific way. But storing data is different to viewing data. When storing data you need the flexibility to adjust your storage to new kind of entries in your database. Relational database systems do this via tables. Storing data in tables will higher the risk for many conversions and complex table joins.

Mittwoch, 21. Juli 2010

android API source JAR

Howto

A few days ago I installed the android SDK and the ADT plugin for eclipse. When I started playing with the android API I was a little disappointed. I was missing the source attachment for the android.jar which contains the android API. I was searching the web for the android sources jar and found a guide how to create an android sources jar [1].

I've adapted the sources howto and created my own android sources jar for android platform 7 aka android 2.1. You can download it here: http://www.bughome.de/android/platforms/android-7/android-src.jar

I'm pretty sure that the android-src.jar is missing some sources, because the source jar is smaller than the binary jar. Comment me if you find the missing sources.

References

  1. http://androidforums.com/android-developers/1045-source-code-android-jar.html