How does Android Substrate work? - java

In Android world, there are two popular injection/hijack/hooking frameworks :
Xposed and Android Substrate
The Xposed's mechanism is described in a development tutorial.
The author of Android Substrate compared the two frameworks in FAQ and here, but he didn't say how it works.
I just wonder:
How does Android Substrate work?

Jay Freeman(Saurik) in this post(http://www.cydiasubstrate.com/id/34058d37-3198-414f-a696-73e97e0a80db/) talks about the deference in xposed and substrate and also talks about how they deffer in the way they work its a great read.

Although the similarities are vast between the two frameworks, the actual implementation of hooking processes etc differs enough for you to be able to have both frameworks running side by side.
However, the biggest benefits that I can see for Substrate over XPosed, is the fact that XPosed removes the Java Security Model, while Substrate preserves it, as well as having the ability to wait for a class to be loaded before hooking a method as described in the following quote from #xmllmx answer regarding "Orthogonality";
To make this easier, Xposed provides a set of helpers for common use cases: you can hook when the VM starts, when Zygote takes control, when a particular package is loaded, or when a command line application is executed. You need to know which of these to use, and it is still unclear how you'd hook a class loaded via a dyamic runtime-created class loader (such as against downloaded code).
Substrate instead does away with all of this, thanks to MS.hookClassLoad, an API it provides that allows you to wait for particular classes to be loaded from any class loader at any time. This allows you to write hooks in a way that is less brittle to changes, less prone to simple mistakes, and less limited by how the developer of the target application decided to load their program's code.
In conclusion, in my opinion anyway, both frameworks a very similar and both viable options in achieving the same goal. The only major weighing fact which separates them is the timescales between when each was released. Where XPosed has been around for much longer and has been tried and tested and put through it's paces over the interim between the release of XPosed an the release of Cydia Substrate for Android, which Jay Freeman (Saurik) openly acknowledges
Regardless of this, Cydia Substrate is a powerful and extremely viable alternative to XPosed. Furthermore, you don't have to only limit yourself to developing for one of these frameworks, because (as mentioned previously) both frameworks can be installed on your device without conflict, so developing for both or simply trying it, will in no way hinder you from using your currently install XPosed Modules, Packages or Extensions.

Related

Should anyone use unit test library methods and classes in live development environment?

This questions looks weird or may be pointless at all.
Recently, I was reviewing some code in java language where the developer used one of the methods from a unit testing library "org.easytesting".
Example: He was using a method "Strings.isNullOrEmpty" of "Strings" class from this library to verify the non-nullability of some values and was using other classes/methods at other places in the code.
I know a library is developed to make our life easier(basic principles of Java) and can be used anywhere/everywhere, but is there a recommendation about using a unit test library in live development code ?
I know using it won't led to a compatibility issue because unit test cases are executed always.
I searched at many places may be I'm missing a good word to search.
It could be argued that a unit-test library is just a library, but I don't see it like this.
First, the purpose of a unit-test library is to be used in code that is not production code. Which means, that certain quality criteria relevant for production code might not be met. For example, if there is a bug in a unit-test library it is annoying, but normally does not harm the production code. Or, performance may not be quite as relevant, thread safety and so on. I don't want to say that the popular unit-testing frameworks are of a bad quality. But, the developers of these libraries have all the right in the world to take design decisions based on the assumption that their code will not be part of production code.
Secondly, using a library should be done in accordance to the philosophy of the respective library. For example, if you use a certain gui library, this brings implications on the way event handling is done in your application. And, unit-testing frameworks come under the assumption that the framework is in control of the executable (from the test runner). Therefore, all functions from that library may depend on the test runner being set up and running. If some function from the library does not have this dependency, that is an implementation detail which may change with a new version of the library.
Finally, code should communicate intent. That includes includes (pun intended). It was not the intent of the developer to write unit-testing code, but that would be what the inclusion of a unit-testing library would communicate.
Considering that there are other, production-oriented libraries out there which check if a string is empty or null, any use of the testing framework's method should be treated as a strong code smell and identified in code reviews.
In the future, this testing library may introduce a change in other parts which make running it in production either prohibitively expensive or insecure, as the code running through this empty or null check could be leveraged as an area of attack. Worse, if your team wants to pivot away from this testing framework, you now have to change production code which many teams would be reluctant to do if all they're doing is changing test dependencies.
Without looking specifically at test libraries, here's an answer to this more general question:
Should you use the general-programming utility classes that are provided by any framework or library? (For example, should you use the StringUtils/CollectionUtils/etc provided by a web/UI/logging/ORM/... framework).
The arguments by the other answers are still mostly valid even in this more general case. Here are some more:
These utilities have been developed specifically for use by the framework. They likely only contain very specific methods with narrow use cases (those that are actually required by the framework) and nothing more. They might be optimized for specific internal behavior of the framework and not for general purposes.
Framework developers may introduce breaking changes without much thought, since they don't expect many users outside of their framework.
It would be alarming to see imports from e.g. a UI library in your back end code, it looks like code smell.
In modular projects, you wouldn't want to introduce additional dependencies to the framework's utilities (again, a dependency to an UI framework from you back end modules is code smell). It would also add a bunch of unnecessary transitive dependencies that may aren't even compatible with other dependencies.
So I would say generally you shouldn't use the utilities of frameworks, except in those places where you are actually working with those frameworks. But even then you should consider using Apache Commons or Guava etc. for consistency.
Now you could also replace the terms UI and back end with test and production in the last two points. Test frameworks are also special in the sense that you usually don't include them as run-time dependency. So you would need to change the scope of the test dependencies to make them available at run-time. You should avoid this for the reasons given in the last point.

Android: Replacement for missing Security architecture

I'm developing an Android application that allows plugins to run via OSGi (using Apache Felix). However, I want to enforce certain security aspects on a per-plugin basis. I.e. which Plugin is allowed to to access the filesystem. Also, I want to prevent reflection in general.
All the tutorials I've found so far refer to using OSGi security in combination with loading the standard Java SecurityManager. However, the latter doesn't exist on Android thus rendering the whole approach completely unusable.
I don't want to go as far as to use Bytecode Manipulation to prevent things from happening in the various plugins, as this would be both cumbersome and I could always miss something.
So, does anyone have any idea, how I could get a security architecture similar to standard Java security on Android?
Most importantly, how can I prevent reflection from being used?
But also, is there a way to prevent I/O access i.e. on a per thread basis?
Unless you are willing to modify the core OS, you can't. Android standard permissions can restrict access to files and APIs, but there is nothing to stop reflection. The OS uses it, so you if you block it completely things will break. Blocking by UID, process might be doable, but you'd have to modify the core Java API's implementation (in libcore).
A sandbox in a sandbox! It does not sound as a particular attractive idea for performance reason but as far as I know, the SecurityManager can still be set. So you could just set your own security manager and catch the checks as they go. However, this discussion indicates that the Android team feels not very confident about the quality of the base libs. (I would probably have removed all checks to gain some performance, but I nowadays feel the whole idea of in-process security is a solution that is simple, clear and wrong ...)

Update Java code during runtime

about a year ago I stumbled across a nice feature in Java that I cannot for the life of me find again.
Through some magic interface it was apparently possible to declare some classes or functions replaceable during runtime.
I found a nice example guide of someone who ran a simple little program that printed a certain message, he then updated the program using a method I cannot remember anymore and all of a sudden the program had replaced that old print function with a new one.
I've tried looking through the Java API to spark my memory as well as googling but without success. Can anyone here help?
Various app containers can do this.
Basically you'd need to reload the class in a new ClassLoader (unless you're talking about doing this under the debugger, in which case there are completely different APIs available).
In my opinion, this kind of thing is rarely worth the hassle: designing everything so that it can be reloaded is considerably harder than designing it so it can be completely restarted in a new process. It's also easier to be sure exactly what code is running if there's only ever one version loaded in the process.
It's a neat thing to be able to demo, but for most applications it's not worth it. All in my opinion, of course :)
Note that one notable exception is the ability to reload web UI layers without restarting the container: that can make life much easier.
The HotSwap technology was added to Java 1.4 and enable class file replacement at run-time. The feature is provide through the redefineClasses method of the instrumentation package. I think you can also do that through the JPDA interface.
Here is also a reference to what I believe is the research paper that describe the HotSwap mechanism first:
Towards Flexible and Safe Technology for Runtime Evolution of Java Language Applications
Otherwise you can use Classloader, as the other mentionned, but it only provides dynamic class loading, not replacement. The same class loaded twice will be considered as two different types. Combined with interface and/or a bit of reflection, it can however provide ways to update the application at run-time.
Here is a reference to an awesome paper about class loader, and there usage:
Dynamic Class Loading in the Java Virtual Machine
I won't expand on whether this is good or bad, because it was not your question, but I think it's great to have support for run-time software evolution -- too bad that JSR-117 never made it!
This is typically the kind of functionality I gladly leave to infrastructure as it is difficult to get right and easy to get wrong. As Jon mentioned above, most applications do not need it and for those that need it infrastructure is available.
Most application servers allow hot deployment nowadays, and equally most application servers are embeddable and allow them to be stripped down to remove features you do not need.
If it mainly for development, you should look a JRebel which provides this functionality transparently. I've heard they are working on a runtime solution, but I do not know if it is ready for primetime yet.
If you are really motivated to get this to work, then consider using OSGi. It has a steep learning curve, but once you grok it, it does most things right and works very well. I found the pax tools a good starting point but the eclipse toolchain also has good support for it.

How to deal with the most common classes missing on J2ME

I'm trying to code an application which runs un different java platforms like J2SE, J2ME, Android, etc. I already know that I'll have to rewrite most of the UI for each platform, but want to reuse the core logic.
Keeping this core portable involves three drawbacks that I know of:
Keeping to the old Java 1.4 syntax, not using any of the nice language features of Java 5.0
only using external libraries that are known to work on those platforms (that is: don't use JNI and don't have dependencies to other libs which violate this rules)
only using the classes which are present on all those platforms
I know of ways to overcome (1): code in 5.0 style and automatically convert it to 1.4 (retroweaver - haven't tried it yet, but seems ok).
I think (2) is a problem that I just have to accept.
Now I'd like to know what's the best workarround for (3), especially collection classes, which I miss the most. I can think of those:
Most programmers I know just don't use Set, Map, List, etc. and fallback to Vector and plain Arrays. I think this makes code ugly in the first place. But I also know that the right choice between TreeSet/Hashset or LinkedList/ArrayList is crucial for performance, and always using Vector and Arrays can't be right.
I could code my own implementations of that classes. This seems to be reinventing the wheel, and I think I could not do it as good as others have done.
Since Java is open source, I could grab the sourcecode of the J2SE Collections framework and include into my application when building for J2ME. I don't know if this is a good idea, though. Pherhaps there are good reasons not to do this.
Maybe there already are libraries out there, which rebuild the most important features of the collections framework, but are optimized for low end systems, pherhaps by not implementing functionality that is used infrequently. Do you know any?
Thanks for your answers and opinions!
Edit: I finally found a (complex, but nice) solution, and I thought by providing my own answer and accepting it, the solution would become visible at the top. But to the contrary, my answer is still at the very bottom.
J2ME is brutal, and you're just going to have to resign yourself to doing without some of the niceties of other platforms. Get used to Hashtable and Vector, and writing your own wrappers on top of those. Also, don't make the mistake of assuming that J2ME is standard either, as each manufacturer's JVM can do things in profoundly different ways. I wouldn't worry much about performance initially, as just getting correctness on J2ME is enough of a challenge. It is possible to write an app that runs across J2ME, J2SE and Android, as I've done it, but it takes a lot of work. One suggestion that I'd have is that you write the core of your application logic and keep it strictly to java.lang, java.util and java.io. Anywhere where you're going to be doing something that might interact with the platform, such as the file system or network, you can create an interface that your core application code interacts with, that you have different implementations for the different environments. For example, you can have an interface that wraps up HTTP stuff, and uses javax.microedition.io.HttpConnection with J2ME and java.net.HttpURLConnection on Android. It's a pain, but if you want to maintain an app running on all three of those environments, it can get you there. Good luck.
It's been a while since I asked this question, and I while since I found a nice, working solution for the problem, but I had since forgotton to tell you.
My main focus was the Java Collections Framework, which is part of the java.util package.
I've finally taken the source code of Suns Java 6.0 and copied all the classes that belong to the Collections framework into a project of my own. This was a Java 6.0 project, but I used the jars from J2ME as classpath. Most of those classes that I copied depend on other J2SE classes, so there are broken dependencies. Anyway, it was quite easy to cut those depensencies by leaving out everything that deals with serialization (which is not a priority for me) and some minor adjustments.
I compiled the whole thing with a Java 6 compiler, and retrotranslator was used to port the resulting bytecode back to Java 1.2.
Next problem is the package name, because you can't deliver classes from java.util with a J2ME application and load them - the bootstrap class loader won't look into the applications jar file, the other bootloaders aren't allowed to load something with that package name, and on J2ME you can't define custom classloaders. Retrotranslator not only converts bytecode, it also helps to change name references in existing bytecode. I had to move and rename all classes in my project, e.g. java.util.TreeMap became my.company.backport.java.util.TreeMap_.
I was than able to write actual J2ME application in a second Java 6.0 project which referenced the usual java.util.TreeMap, using the generic syntax to create type-safe collections, compile that app to Java 6.0 byte code, and run it through retrotranslator to create Java 1.2 code that now references my.company.backport.java.util.TreeMap_. Note that TreeMap is just an example, it actually works for the whole collections framework and even for 3rd party J2SE Jars that reference that framework.
The resulting app can be packaged as a jar and jad file, and runs fine on both J2ME emulators and actual devices (tested on a Sony Ericsson W880i).
The whole process seems rather complex, but since I used Ant for build automation, and I needed retranslator anyway, there only was a one-time overhead to setup the collection framework backport.
As stated above, I've done this nearly a year ago, and writing this mostly from the top of my head, so I hope there are no errors in it. If you are interested in more details, leave me a comment. I've got a few pages of German documentation about that process, which I could provide if there is any demand.
We faced exactly this situation in developing zxing. If J2ME is in your list of targets, this is your limiting factor by far. We targeted MIDP 2.0 / CLDC 1.1. If you have a similar requirement, you need to stick to Java 1.2. Java 1.4 language features are definitely not present (like assert) and in general you won't find anything after 1.2 in J2ME.
We did not use external libraries, but, you could package them into your deployed .jar file with little trouble. It would make the resulting .jar bigger, and that could be an issue. (Then you can try optimizers/shrinkers like ProGuard to mitigate that.)
I did end up reimplementing something like Collections.sort() and Comparator since we needed them and they are not in J2ME. So yeah you might consider doing this in cases, though only where necessary.
We used Vector and Hashtable and arrays since there is no other choice, really, in J2ME. I would just use them unless you have a reason not to, and that would be performance I guess. In theory JVM makers are already optimizing their implementation but that doesn't mean you couldn't make a better one... I guess I would be surprised if it is worth it in the vast majority of cases. Just make sure you really need to do this before putting in the effort.
To answer part of your question another collections library would be Javolution which can be built for j2me.

Logic (if any) behind Google App Engine excluding standard JDK 1.6 APIs

It looks like GAE has chosen a subset of JDK 1.6 classes, as per:
Google App Engine JDK white list
which is very unfortunate as one gets class linkage errors all over the place with most common java libraries that deal with data binding, reflection, class loading and annotations. Although some omissions may be for deprecated or legacy things, there are others that are not. My specific concern is with streaming pull parsers (javax.xml.stream.*) which was just added to JDK 1.6 after a long delay (API was finalized at about same time as JDK 1.4). Omitting this makes it harder to do scalable high-performance xml processing.
Problem as I understand is that not only are classes missing, but they can not even be added because of security constraints.
So: this is an open-ended philosophical question that probably just GAE devs could answer for sure but... why are some APIs dropped from standard JDK 1.6, seemingly arbitrarily?
UPDATE:
Quick note: thanks for answers. For what it's worth I really do not see how security would have anything to do with not including javax.xml.stream.
Security aspects are relevant for great many other things (and I don't need threads, for example, and can see why they are out), so it's understandable boilerplate answer; just not applicable for this one.
Stax API is just a set of interfaces and abstract for crying out loud. But more importantly, it has exactly the same ramifications as including SAX, DOM and JAXP interfaces -- which are included already!
But it looks like this issue has been brought to attention of google devs:
discussion on whitelisting Stax API
so here's hoping that this and similar issues can be resolved swiftly.
GAE is run in a hosted environment with untrusted (and potentially malicious) clients, who often are given access for free.
In that type of environment, security is a very high concern, and APIs which have filesystem access get very heavy scrutiny. I think thats why they've chosen to start pretty conservatively in terms of what they allow.
It wouldn't surprise me at all if more classes find their way into the whitelist as security issues are addressed (and based on demand), though.
But I wouldn't even expect to get threading tools available, eg.
It's extremely doubtful that these things were dropped arbitrarily. GAE runs in an extremely security-sensitive environment, and the chances are good that an internal audit of the class libraries found some risks that Google was not willing to take.
As for your high-performance streaming XML parsers, you could try to find an appropriate library (jar file). Unless it relies on threads or file access (or black-listed API), it should work just as well as the one in the JDK.
There are a lot of (rather complex) libraries that work on GAE.

Categories

Resources