Develop Reference

Apache Flink Kerberos configuration in custom deserializer for Hortonworks Schema Registry

I'm trying to connect to Hortonworks Schema Registry from a custom deserializer for a Kafka topic using Apache Flink.
I am able to read messages from secured Kafka by setting sasl.jaas.config property on Kafka consumer. But when I add a custom deserializer that uses Hortonworks Schema Registry to get the deserialization schema, the service response with 401 HTTP Code: Authentication required. I've Kerberos enabled on this service.
What I 've tried:
Set java.security.auth.login.config on deserializer implementation:
System.setProperty("java.security.auth.login.config", "/path/to/jaas.conf");
Launch application with dynamic Java options:
flink run [...] -yD env.java.opts="-Djava.security.auth.login.config=/path/to/jaas.conf" myApplicatio.jar
Setting env.java.opts in flink-conf.yaml:
env.java.opts: -Djava.security.auth.login.config=/path/to/jaas.conf
Neither of these options worked for me.
The content of the jaas.conf to authenticate against HW Schema registry is:
RegistryClient {
com.sun.security.auth.module.Krb5LoginModule required
principal="flink#mydomain.com"
keytab="/etc/security/keytabs/flink.keytab"
useKeytab=true
useTicketCache=true
renewTicket=true
serviceName="flink";
};
Could someone help on this?
Update: I whing something is missing in the server since a curl request shows Internal server error on Spenego negotiation, The traceback on the server logs is:
ERROR [2019-10-10 10:49:46.099] [dw-26402 - GET /api/v1/schemaregistry/schema] c.h.r.c.GenericExceptionMapper - Got exception: [NotFoundException] / message [HTTP 404 Not Found]
javax.ws.rs.NotFoundException: HTTP 404 Not Found
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:323)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:34)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at com.hortonworks.registries.schemaregistry.webservice.RewriteUriFilter.doFilter(RewriteUriFilter.java:77)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at com.hortonworks.registries.auth.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)
at com.hortonworks.registries.auth.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:311)
at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:265)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:241)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:69)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678)
at java.lang.Thread.run(Thread.java:745)
Thank you

GWT: Class cast exception while dispatching incoming RPC call

I am developing a web application and get this error but can't make sense of it (I've looked at similar questions but the answers there don't seem to apply to my problem)
Can anyone make sense of this?
Please help me.
[WARN] Exception while dispatching incoming RPC call
java.lang.ClassCastException: java.lang.NoClassDefFoundError cannot be cast to java.lang.Exception
at net.sf.gilead.gwt.PersistentRemoteService.processCall(PersistentRemoteService.java:177)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
at org.realityforge.gwt.cache_filter.GWTGzipFilter.doFilter(GWTGzipFilter.java:57)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at ... (more eclipse.jetty...)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:748)
[ERROR] 500 - POST /webproj/login (127.0.0.1) 57 bytes
Request headers
Host: (...)
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: text/x-gwt-rpc; charset=utf-8
(leaving some stuff out)
Connection: keep-alive
Response headers
Content-Type: text/plain

Jackson deserialisation error when upgrading spring-cloud-netflix from 1.3.0.M1 -> 1.3.0.RELEASE

I am getting jackson deserialisation error when I upgraded spring cloud netflix from 1.3.0.M1 -> 1.3.0.RELEASE.
The Post request that I hit is
Received [POST /wms-mdm/catalog HTTP/1.1
Host: localhost:8093
Content-Type: application/json
Origin: http://localhost:8093
Accept-Encoding: gzip, deflate
Connection: keep-alive
Accept: */*
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Safari/602.1.50
Referer: http://localhost:8093/wms-mdm/swagger-ui.html
Content-Length: 99
Accept-Language: en-us
With Data
{
"clientCatalogId": "7",
"shortDescription": "Dummy Catalog",
"longDescription": "Dummy Catalog"
}]
The error I am getting is:
com.fasterxml.jackson.databind.JsonMappingException: Could not find creator property with name 'id' (in class com.gor.platform.mdm.core.pojo.Catalog)
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.DeserializationContext.reportMappingException(DeserializationContext.java:1234) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.addBeanProps(BeanDeserializerFactory.java:551) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:226) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:141) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:403) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.deser.DeserializerCache.hasValueDeserializerFor(DeserializerCache.java:191) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.DeserializationContext.hasValueDeserializerFor(DeserializationContext.java:421) ~[jackson-databind-2.8.7.jar:2.8.7]
at com.fasterxml.jackson.databind.ObjectMapper.canDeserialize(ObjectMapper.java:2713) ~[jackson-databind-2.8.7.jar:2.8.7]
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.canRead(AbstractJackson2HttpMessageConverter.java:162) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:195) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.readWithMessageConverters(RequestResponseBodyMethodProcessor.java:150) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:128) [spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:121) [spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
When I downgrade netflix cloud to old version things works fine. Anyone can help?

Adding explicit dependency for jackson-databind 2.8.8 solved the issue. Looks like some problem with 2.8.7

Jersey Client multi-part/form-data message body writer exception

I feel like I am very close but missing a piece of the puzzle here. I am getting an exception about message body writer. Any help is appreciated.
Server side code
#POST
#Path("/{id}/documents")
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Produces(MediaType.APPLICATION_JSON)
public Response uploadAndAddNewDocument(
#FormDataParam("file") final InputStream fileInputStream,
#FormDataParam("file") final FormDataContentDisposition fileDetail,
#FormDataParam("file") final FormDataBodyPart fileBodyPart,
#FormDataParam("model") final SomeModel model,
#FormDataParam("model") final FormDataBodyPart modelBodyPart) throws WebApplicationException {
// Here model is received as application/json
// Code to process model and uploaded file
// This works when posted using POSTMAN
}
Server code works with following POSTMAN request
POST /someresource/123/documents HTTP/1.1
Host: localhost:8080
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Cache-Control: no-cache
Postman-Token: c6425906-1b1f-39d0-bed6-93db9e9a6ad9
----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="file"; filename="report.xml"
Content-Type: application/xml
<root></root>
----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="doc"
Content-Type: application/json
{}
----WebKitFormBoundary7MA4YWxkTrZu0gW
Jersey Client Code
//Client configuration code
config.getClasses().add(MultiPartWriter.class);
//Client call to server
FormDataMultiPart multiPartFormData = new FormDataMultiPart();
multiPartFormData.bodyPart(new BodyPart("doc", MediaType.APPLICATION_JSON_TYPE));
multiPartFormData.bodyPart(new FileDataBodyPart("file", uploadFile));
return service.path(uri).type(MediaType.MULTIPART_FORM_DATA).post(SomeModel.class, multiPartFormData);
**Error
Mar 03, 2015 11:55:40 PM com.sun.jersey.api.client.RequestWriter$RequestEntityWriterImpl <init>
SEVERE: A message body writer for Java class com.sun.jersey.multipart.FormDataMultiPart, and Java type class com.sun.jersey.multipart.FormDataMultiPart, and MIME media type application/octet-stream was not found
Mar 03, 2015 11:55:40 PM com.sun.jersey.api.client.RequestWriter$RequestEntityWriterImpl <init>
SEVERE: The registered message body writers compatible with the MIME media type are:
application/octet-stream ->
com.sun.jersey.core.impl.provider.entity.ByteArrayProvider
com.sun.jersey.core.impl.provider.entity.FileProvider
com.sun.jersey.core.impl.provider.entity.InputStreamProvider
com.sun.jersey.core.impl.provider.entity.DataSourceProvider
com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider
*/* ->
com.sun.jersey.core.impl.provider.entity.FormProvider
com.sun.jersey.core.impl.provider.entity.MimeMultipartProvider
com.sun.jersey.core.impl.provider.entity.StringProvider
com.sun.jersey.core.impl.provider.entity.ByteArrayProvider
com.sun.jersey.core.impl.provider.entity.FileProvider
com.sun.jersey.core.impl.provider.entity.InputStreamProvider
com.sun.jersey.core.impl.provider.entity.DataSourceProvider
com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$General
com.sun.jersey.core.impl.provider.entity.ReaderProvider
com.sun.jersey.core.impl.provider.entity.DocumentProvider
com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider
com.sun.jersey.core.impl.provider.entity.SourceProvider$SourceWriter
com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider$General
com.sun.jersey.core.impl.provider.entity.XMLListElementProvider$General

there are few things you need to check
add mimepull.jar to your lib
or with Maven
<dependency>
<groupId>org.jvnet.mimepull</groupId>
<artifactId>mimepull</artifactId>
<version>1.9.5</version>
</dependency>
And if you are working with file, make sure you send at the header the
content-length
content-type
accept-encoding

Initializing Hibernate with GWT problem

I'm trying to use Hibernate with GWT (using Gilead) but I got the following error
[WARN] /hibernate/greet
java.lang.NoClassDefFoundError: Could not initialize class net.jaffa.hibernate.server.NewHibernateUtil
at net.jaffa.hibernate.server.GreetingServiceImpl.<init>(GreetingServiceImpl.java:26)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.mortbay.jetty.servlet.Holder.newInstance(Holder.java:153)
at org.mortbay.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:339)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:463)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
[ERROR] 500 - POST /hibernate/greet (127.0.0.1) 3632 bytes
Request headers
Host: 127.0.0.1:8888
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://127.0.0.1:8888/hibernate/hosted.html?hibernate
X-GWT-Permutation: HostedMode
X-GWT-Module-Base: http://127.0.0.1:8888/hibernate/
Content-Type: text/x-gwt-rpc; charset=utf-8
Content-Length: 175
Pragma: no-cache
Cache-Control: no-cache
Response headers
Content-Type: text/html; charset=iso-8859-1
Content-Length: 3632
This is still a new project, just I'm trying to do proof of concept.
Thanks
Feras

java.lang.NoClassDefFoundError: Could not initialize class n.j.h.s.NewHibernateUtil
This error means that the class is found but its initialization failed. Possible causes include
some dependencies are unsatisfied ~or~
an exception is thrown when executing a static initialization block in the class.
In the particular case of a typical HibernateUtil class, I'd bet on the second option. Extract the code of the static initialization block and run it / debug it in a regular main, outside the container (and maybe try to improve the error reporting in case of problem).

java.lang.NoClassDefFoundError means the runtime could not locate the class in its classpath.
It seems you are missing a jar which contains the class net.jaffa.hibernate.server.NewHibernateUtil .