I'd like to trace a java application at runtime to log and later analyze every its behaviour.
Is there a possibility to hook into a java application to get runtime information like method calls (with parameters and return values) and the status of an object (i.e. its attributes and whose values)?
My goal is to get a complete understanding of the applications behaviour and how it deals with the data.
If you need highly customized logging and runtime processing, one alternative to profilers is to use aspects and load-time weaving.
We use AspectJ in this way to capture and log the authentication information for users who call a number of low-level methods for debugging purposes and to undo mistaken changes.
Use a profiler. For example JProfiler or one from this overview of opensource java profilers. Whenever I had to find deadlocks for example, these tools were priceless...
In Netbeans the profiler exist and work properly for use it see http://profiler.netbeans.org/
Maybe have a look at Glassbox a troubleshooting agent for Java applications that automatically diagnoses common problems. From Glassbox - Automated monitoring and troubleshooting using AOP:
Glassbox deploys as a war file to your
appserver and then uses AspectJ load
time weaving to monitor application
components and other artifacts, in
order to identify problems like excess
or failed remote calls, slow queries,
too many database queries, thread
contention, even what request
parameters caused failures. All this without
changing the code or the build process.
(...)
Glassbox monitors applications non-invasively by using aspects to track component interactions. We also monitor built-in JMX data, notably on a Java 5 VM we sample thread data (every 100 ms by default). As a request is processed, we summarize noteworthy events such as where time was spent and what parameters were involved in making things slow or fail. We also detect higher-level operations (such as Struts actions or Spring controllers) that we use to report on. Our AJAX Web client then provides summaries of status by operation on the machines being monitored and we generate a more detailed analysis on request. Glassbox allows monitoring clusters of servers: the Web app uses JMX Remote or direct RMI to access data from remote servers. We also provide JMX remote access to the lower-level summary statistics.
It's a nice application, give it a try.
Related
I am currently working on a legacy application where database calls are kind of scattered all over. I need to execute some logic linked to security (business) every time some sort of DML is executed. For this, I am thinking of using a java agent and intercepting the calls and subsequently executing the business logic.
The issue is that this agent needs to be secured and I need to ensure that there is no way that a different agent developed with a similar but different logic is loaded.
Is there any kind of mutual authentication possible between the application and the java agent which would ensure that a wrong agent is in no way able to be loaded
Is there any kind of mutual authentication possible between the application and the java agent which would ensure that a wrong agent is in no way able to be loaded.
No. The application has (almost) no knowledge of the agent. Certainly there is no way to for the application to properly validate the agent.
I guess, you could design a "protocol" where the application only works if an agent calls a particular application method in a particular way. However that could be circumvented by reverse engineering the application or a real agent, and using that knowledge to write a bad agent that imitates the required behavior.
But I think that you are going about this the wrong way. There are many better (simpler, cleaner, more efficient) ways to inject behavior into a Java application. And I think I can detect that part of your motivation is that you want to modify the legacy application as little as possible. And that is the primary motivation for your complicated agent-based approach.
(My reaction to that is that you are likely to spend more effort on the agent stuff than you would save by not modifying the legacy app. And the result will be much harder to maintain.)
I also suspect that your requirement for mutual authentication of the application and the interceptors (however they are implemented) is not really necessary. A consequence of using agents rather than a fundamental requirement.
If I was doing this and I was this concerned about protecting against (insider) attempts to subvert the business rules, I would:
Choose an alternative mechanism
Implement the new code and modifications to the legacy app as required.
Audit the main application and interceptor logic
Join the two parts together (depending on how the mechanism work)
Put the combined system onto a secured machine or container.
The Java attach API is already secured in the way that an agent must either:
be specified on the command line.
be attached from a JVM running by the same OS user.
To establish both cases, you must own privileges that can already escalate beyond the privileges of the agent running within the JVM process. For these reasons, it should not be necessary to validate your agent from within the JVM.
In theory you could write a Java agent that instruments the instrumentation API and make sure this agent is attached first. The instrumentation API's instrumentation could then check the jar file of the attached agent prior to its attachment and compare it to some known seed, for example. If this seed does not match, you could fail the agent's initialization.
I am developing a product using microservices and am running into a bit of an issue. In order to do any work, I need to have all 9 services running on my local development environment. I am using Cloud Foundry to run the applications, but when running locally I am just running the Spring Boot Jars themselves. Is there anyway to setup a more lightweight environment so that I don't need everything running? Ideally, I would only like to have the service I am currently working on to have to be real.
I believe this is a matter of your testing strategy. If you have a lot of micro-services in your system, it is not wise to always perform end-to-end testing at development time -- it costs you productivity and the set up is usually complex (like what you observed).
You should really think about what is the thing you wanna test. Within one service, it is usually good to decouple core logic and the integration points with other services. Ideally, you should be able to write simple unit tests for your core logic. If you wanna test integration points with other services, use mock library (a quick google search shows this to be promising http://spring.io/blog/2007/01/15/unit-testing-with-stubs-and-mocks/)
If you don't have already, I would highly recommend to set up a separate staging area with all micro-services running. You should perform all your end-to-end testing there, before deploying to production.
This post from Martin Fowler has a more comprehensive take on micro-service testing stratey:
https://martinfowler.com/articles/microservice-testing
It boils down to a test technique that you use. Here my recent answer in another topic that you could find useful https://stackoverflow.com/a/44486519/2328781.
In general, I think that Wiremock is a good choice because of the following reasons:
It has out-of-the-box support by Spring Boot
It has out-of-the-box support by Spring Cloud Contract, which gives a possibility to use a very powerful technique called Consumer Driven Contracts.
It has a recording feature. Setup your Wiremock as a proxy and make requests through it. This will generate stubs for you automatically based on your requests and responses.
There are multiple tools out there that let you create mocked versions of your microservices.
When I encountered this exact problem myself I decided to create my own tool which is tailored for microservice testing. The goal is to never have to run all microservices at once, only the one that you are working on.
You can read more about the tool and how to use it to mock microservices here: https://mocki.io/mock-api-microservices. If you only want to run them locally, it is possible using the open source CLI tool
It can be solved if your microservices allow passing metadata along with requests.
Good microservice architecture should use central service discovery, also every service should be able to take metadata map along with request payload. Known fields of this map can be somehow interpreted and modified by the service then passed to next service.
Most popular usage of per-request metadata is request tracing (i.e. collecting tree of nodes used to process this request and timings for every node) but it also can be used to tell entire system which nodes to use
Thus plan is
register your local node in dev environment service discovery
send request to entry node of your system along with metadata telling everyone to use your local service instance instead of default one
metadata will propagate and your local node will be called by dev environment, then local node will pass processed results back to dev env
Alternatively:
use code generation for inter-service communication to reduce risk of failing because of mistakes in RPC code
resort to integration tests, mocking all client apis for microservice under development
fully automate deployment of your system to your local machine. You will possibly need to run nodes with reduced memory (which is generally OK as memory is commonly consumed only under load) or buy more RAM.
An approach would be to use / deploy an app which maps paths / urls to json response files. I personally haven't used it but I believe http://wiremock.org/ might help you
For java microservices, you should try Stybby4j. This will mock the json responses of other microservices using Stubby server. If you feel that mocking is not enough to map all the features of your microservices, you should setup a local docker environment to deploy the dependent microservices.
From the terminology point of view and in general, what is the difference between a tracing and a logging ?
Thanks!
Logging is not Tracing!
Logging
When you design a big application, you need to have good and flexible error reporting - perhaps across machines - to collect log data in a centralized way. That is a perfect use case for the Logging Application Block where you configure some remote trace listener, and send the log data to a central log server which stores its log messages in a database, log file or whatever. If you use out-of-process communication, you are limited by the network performance already, which in the best case is several thousand logs/s.
Tracing
Besides Error Reporting, you also need to trace your program flow to find out where the performance bottlenecks are; even more importantly, when an error occurs, you have a chance to find out how you did get there. In an ideal world, every function would have some tracing enabled with the function duration, passed parameters, and how far you did get into your function.
If the context is developing an Observability capability across a distributed architecture, it's common for people to talk about metrics, logs, and tracing. In this context, tracing refers to distributed tracing.
Distributed tracing is a specialised type of telemetry (similar to logging, but different), and is usually produced in a highly automated fashion through instrumentation frameworks. The telemetry is sent from the individual services in the system and aggregated by a central service (a distributed tracer), which is able to piece together telemetry from many individual services into a single trace for each request that entered the system. It can then provide a timeline and graph of how a request moved through the services in the system. The main purposes of distributed traces are to investigate performance degradations, error propagation, and dependency interactions throughout distributed systems.
Whereas tracing in a more traditional monolithic context would typically be looking at tracing individual function calls within an application, distributed tracing is typically only concerned with the interactions between services. Telemetry of function-call-level details is possible but rarely implemented.
For more info about distributed tracing, a good intro can be found at: https://opentelemetry.lightstep.com/tracing/
Trace is the least filtered level of logging. Each logging statement has a level of filtering:
trace
debug
warning
error
severe
For example. if the logging library is configured to log with level warning then all warning, error and severe logging statements will be printing messages to the logging output.
Logging is for performance monitoring too. Does not need to be true that only trace is able to find out where the performance bottlenecks are. Both can work in distributed mode.
I successfully set up JMX monitpring in zabbix.
With the default template, I can get a good idea about the performance of my application.
I now need to monitor the state of the running threads. So find out, weather it is running, paused or interrupted.
In jConsole, I can see this. So I assume, JMX does allow this.
Does anybody know if this can be done within zabbix monitoring? I assume I need to define a discovery rule for the threads, but I cannot find any docs about this.
Thanks for your help, all input appreciated!
There are two aspects to your question: (a) observing the state of the existing threads through JMX and (b) low-level discovery for JMX thread monitoring using Zabbix.
Regarding (a), I am currently not aware of any JMX object that would expose this information. By default JVM provides objects like jmx["java.lang:type=Threading",ThreadCount] and jmx["java.lang:type=Threading",DaemonThreadCount], both of which are included in the default "Template JMX Generic". However, I am not aware of any JMX objects that would provide a more detailed information. Please share if you know some.
Regarding (b), there is a built-in item jmx.discovery, which returns a list of all JMX objects found on the JMX agent. However, with limited filtering abilities and without a possiblity to create an item of the correct type based on {#JMXTYPE} macro, it was mostly meant for internal testing purposes and that is why it is undocumented.
However, you can find an example of jmx.discovery at ZBX-6273. With the recent addition of multiple LLD filters in ZBXNEXT-581, you can try filtering the thread objects of interest. Alternatively, you can wait until ZBXNEXT-1223 is implemented or use the patches and utilities mentioned there in the meanwhile. In particular, the recent addition at https://www.zabbix.org/wiki/Docs/howto/jmx_discovery is useful.
Okay. This is again a question of industry practice.
Tomcat = Web Container
JBoss, WebLogic, etc = Application Servers that have Web Container within (for JBoss, its forked Tomcat)
Spring does not need Application Server like JBoss. If we use enterprise services like JMS, etc we can use independent systems like RabbitMQ, ApacheMQ, etc.
Question is why do people still use JBoss and other Application Serves for purely spring based applications?
What are the advantages Spring can make use of, by using Application Servers? Like object pooling? What specific advantages does Application Server offers? How are those configured?
If not for spring, for what other purposes Application Servers are used for Spring/Hibernate, etc stack? (Use cases)
Actually I would say listening for JMS is probably the best reason for an application server. A stand alone message broker does not fix the problem since you still need a component that's listening for messages. The best way to do this is to use a MDB. In theory you can use Springs MessageListenerContainer. However this has several disadvantages like JMS only supports blocking reads and Spring therefore needs to spin up it's own threads which is totally unsupported (even in Tomcat) and can break transactions, security, naming (JNDI) and class loading (which in turn can break remoting). A JCA resource adapter is free to do whatever it wants including spinning up threads via WorkManager. Likely a database is used besides JMS (or another destination) at which point you need XA-transactions and JTA, in other words an application server. Yes you can patch this into servlet container but that this point it becomes indistinguishable from an application server.
IMHO the biggest reason against application servers is that it takes years after a spec is published (which in turn takes years as well) until severs implement the spec and have ironed out the worst bugs. Only now, right before EE 7 is about to be published do we have are EE 6 servers starting to appear that are not totally riddled with bugs. It gets comical to the point where some vendors do no longer fix bugs in their EE 6 line because they're already busy with the upcoming EE 7 line.
Edit
Long explanation of the last paragraph:
Java EE in a lot of places relies on what's called contextual information. Information that's not explicitly passed as an argument from the server/container to the application but implicitly "there". For example the current user for security checks. The current transaction or connection. The current application for looking up classes to lazily load code or deserialize objects. Or the current component (servlet, EJB, …) for doing JNDI look ups. All this information is in thread locals that the server/container sets before calling a component (servlet, EJB, …). If you create your own threads then the server/container doesn't know about them and all the features relying on this information don't work anymore. You might get away with this by just not using any of those features in threads you spawn.
Some links
http://www.oracle.com/technetwork/java/restrictions-142267.html#threads
http://www.ibm.com/developerworks/websphere/techjournal/0609_alcott/0609_alcott.html#spring-4
If we check the Servlet 3.0 specification we find:
2.3.3.3 Asynchronous processing
Java Enterprise Edition features such as Section 15.2.2, “Web Application Environment” on page 15-174 and Section 15.3.1, “Propagation of Security Identity in EJBTM Calls” on page 15-176 are available only to threads executing the initial request or when the request is dispatched to the container via the AsyncContext.dispatch method. Java Enterprise Edition features may be available to other threads operating directly on the response object via the AsyncContext.start(Runnable) method.
This is about asynchronous processing but the same restrictions apply for custom threads.
public void start(Runnable r) - This method causes the container to dispatch a thread, possibly from a managed thread pool, to run the specified Runnable. The container may propagate appropriate contextual information to the Runnable.
Again, asynchronous processing but the same restrictions apply for custom threads.
15.2.2 Web Application Environment
This type of servlet container should support this behavior when performed on threads created by the developer, but are not currently required to do so. Such a requirement will be added in the next version of this specification. Developers are cautioned that depending on this capability for application-created threads is not recommended, as it is non-portable.
Non-portable means it can may in one server but not in an other.
When you want do receive messages with JMS outside of an MDB you can use four methods on javax.jms.MessageConsumer:
#receiveNoWait() you can to this in a container thread, it doesn't block, but it's like peeking. If no message is present it just returns null. This isn't very well suited for listening to messages.
#receive(long) you can to this in a container thread, it does block. You generally don't wan't to do blocking waits in a container thread. Again not very well suited for listening to messages.
#receive(), this blocks possibly indefinitely. Again not very well suited for listening to messages.
#setMessageListener() this is what you want, your get a callback when a message arrives. However unless the library can hook into the application server this won't be a container thread. The hooks into the application server are only available via JCA to resource adapters.
So yes, it may work, but it's not guaranteed and there are a lot of things that may break.
You are right that you don't really need a true application server (implementing all Java EE specs) to use Spring. The biggest reason people don't use true Java EE apps like JBoss is that then have been slow as #$##% on cold start up time making development a pain (hot deploy still doesn't work that well).
You see there are two camps:
Java EE
Spring Framework.
One of the camps believes in the spec/committee process and the other believes in benevolent dictator / organic OSS process. Both have people with their "agendas".
Your probably not going to get a very good unbiased answer as these two camps are much like the Emacs vs VIM war.
Answer your questions w/ a Spring bias
Because it in theory buys your less vendor lock-in (albeit I have found this to be the opposite).
Spring's biggest advantage is AspectJ AOP. By far.
I guess see Philippe's answer.
(start of rant)
Since #PhilippeMarschall defended Java EE I will say that I have done the Tomcat+RabbitMQ+Spring route and it works quite well. #PhilippeMarschall discussion is valid if you want proper JTA+JMS but with proper setup with Sprig AMQP and an a good transactional database like Postgresql this is less of an issue. Also he is incorrect about the message queue transactions not being bound/synchronized to the platform transactions as Spring supports this (and IMHO much more elegantly with #Transactional AOP). Also AMQP is just plain superior to JMS.
(end of rant)
We are using JBoss over tomcat for the JNDI data sources and pooling.. It makes it so the programmer don't have to know anything about the database but its JNDI name