Develop Reference

Do I need to put some sort of mutual exclusion while multiple threads are writing on the same MulticastSocket?

I have some threads which are writing on the same MulticastSocket (depending by the scheduling, probably can happen than more then one thread is writing on the MulticastSocket at the same time). Do I have to get them write on it one per timer by using some form of locking, or the UDP protocol is doing this implicitly?

It doesn't really have anything to do with UDP. The documentation for MulticastSocket doesn't say it's threadsafe, so you can't assume it's threadsafe. You can't know that it doesn't update internal structures (such as an outbound buffer) which could be damaged by simultaneous access.
If all the threads are using the same instance of MulticastSaocket, you'll want to ensure they don't simultaneously call its methods. You can do that easily enough by synchronizing on the instance:
synchronized (theSocket) {
theSocket.send(/*...*/);
}

How to publish messages to rabbitmq with high tps, multithreading

Since channel is not thread safe, I can either synchronize th channel instance before publish or I create a channel each time I need and close it.
But in my opinion neither of them have a good performance due to cost of locking or create and destory channels.
So how should I publish message to rabbitmq with high tps? Any good pratise on this?

So, first thing first. A channel is not a connection. In RabbitMQ, Channels are the same thing as application sessions, whereas a connection represents an underlying TCP session with the server. This answer explains some of that nicely.
TCP sessions are expensive to create, so they tend to have a lifetime outside the lifetime of any particular worker thread. Channels are extremely cheap to create - all the server does is assign you an integer for your channel identifier and you have a new channel.
Most operations on RabbitMQ close a channel when they fail. This is done because there is no practical consequence of doing so. Would they close the underlying connection, that would cause a lot of problems for the app.
Design Guidance
Pooling would be appropriate for connections, if you really have a lot of processing going on. A discussion on how to do this is really beyond what I can provide in a short answer.
Pooling is absolutely not appropriate for channels. A channel is a lightweight construct that is designed to have a transient lifetime. If it lasts longer than one or two publishes, great. But you should expect that every time you try an operation, there is a possibility it will fail and close the channel. This does not close the underlying connection, but a new channel will have to be reestablished to do anything with the broker.
Consumer lifetimes are tied to channels. When the channel closes, the attached consumer is closed too. Design your consumer objects (worker threads) to be able to get a connection and create a new channel when this happens, and then re-subscribe themselves.
Avoid sharing a channel across threads. One thread = one channel.
While I don't have any particular experience with the Java client, I don't believe locks should be necessary, and I would certainly hope that the implementation doesn't do something screwy to make Channels anything other than lightweight.
If you're writing your own protocol implementation library (not necessary, but also not a bad idea if you need fine-grained control), allocate one thread to manage each connection. Don't do reads and writes to the TCP socket in parallel, or you'll break the protocol.
Regarding the Java client, I think you can assume that channel operations (reads and writes, etc.) are NOT thread-safe, which is why you want to stick to one thread/one channel paradigm. I think you can assume that creating channels (a connection operation) is thread-safe.

You should use a pool.
For instance, use Apache's Generic Object Pool and provide an implementation for opening, closing and checking connections. When you need to publish a message, you borrow a channel from the pool, use it, and return it.

Thread pool when serving multiple clients with blocking methods

I am developing a webserver in java that will provide websocket communication to its' clients. I have been proposed to use a thread pool when dealing with many clients because it is a lot more time efficient than to use one thread per client.
My question is simply, will Javas ExecutorService, newFixedThreadPool be able to handle a queue of runnable tasks with thread blocking methods being called inside of them?
In other words i guess i am wondering if this thread pool is asynchronous?
The reason i am asking is that i have tried using a newFixedThreadPool with, lets say, 2 threads. Then when i connect 3 clients to the server, i can only receive commands from the first two. But i guess i could be doing something wrong, thats why i am asking.
The runnable tasks are also in an infinite while loop (only ends when client disconnects).

Well, it depends on your implementation. The easiest case is having clients keeping their thread active until the disconnect (or get kicked out because of a timeout). In this case, your thread pool isn't very efficient. I'll only re-use disconnected users' threads instead of creating new one (which is good, but not really relevant).
The second case would be activating the threads only when needed (let's say when a client sends or receives a messages). In this case, you need to remember the server-side (keeping an id for example), in order to be able to sever the thread connection when they don't need them, and re-establish it when they do. In order to do that, you must keep the sockets somewhere, but unbound to any specific thread.
I actually didn't code that myself but I don't see why it would work as this is the mechanism used for websites (i.e. HTTP protocol)

Is Java socket multi-thread safe?

If I have multiple Java threads writing to the same Socket instance simultaneously, will that affect the integrity of the objects that are read from the same socket? I.e., whether the contents of the objects will be messed up etc. It's fine for the ordering of objects to be random.

In general, there are no guarantees. Bits of different objects could well end up getting interleaved on the wire, rendering the result indecipherable. Therefore, you need to provide external synchronization.
It is interesting to note that even a single socket write at the OS level is not necessarily atomic. For further discussion, see Is it safe to issue blocking write() calls on the same TCP socket from multiple threads? and Be careful with the sendmsg() family of functions.

If I have multiple Java threads writing to the same Socket instance
simultaneously
You will be writing to the same OutputStream from multiple threads.
What makes you think that it is a good idea without synchronization? If you started writing to a file from multiple threads simultaneously without synchronization would you expect the file to contain anything meaningfull?

notify a thread in the client that was send on wait() in the server

I have a big problem to resolve and I don't know how to resolve it. In a thread server I call on a object the method wait() after using ObjectOutputStream send the object to a client thread that is connected to the server. When the object arrives on the other side I call on the object the method notify() but the object doesn't wake up from his state of waiting. This is caused because the object sent it isn't the same. How can I resolve this big issue?

Irrespective of whether the client and server are on two different machines / JVMs, the object you get after serialization - deserialization is different than the original object.
When you call "wait()" on one object, the thread blocks until some other thread call notify() on the SAME object.
You are having two different objects here and so, wait-notify mechanism won't work.
You have to send some acknowledgement to the server using network communication ( sockets may be).

If I understand you correctly, you are serializing an object on the server, sending it to a client, calling notify() on the client, and expecting that to affect the server. This won't work because they are on different computers, or at least different JVMs. The only way you can send a message back to the server is through your socket.

I suggest to read the documentation about synchronization.