Develop Reference

kafka to hdfs with confluent source code

For the requirement of my project, I need to build a class from the confluent java code to write data from kafka topic to the hdfs filesystem.
It is actually working in CLI with connect-standalone, but I need to do the same thing with the source code which I built successfully.
I have a problem with SinkTask and hdfsConnector classes.
An exception is showing up in the put method.
Here below is my class code:
package io.confluent.connect.hdfs;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.connect.errors.ConnectException;
import org.apache.kafka.connect.sink.SinkConnector;
import org.apache.kafka.connect.sink.SinkRecord;
import org.apache.kafka.connect.sink.SinkTaskContext;
import io.confluent.connect.avro.AvroData;
import io.confluent.connect.hdfs.avro.AvroFormat;
import io.confluent.connect.hdfs.partitioner.DefaultPartitioner;
import io.confluent.connect.storage.common.StorageCommonConfig;
import io.confluent.connect.storage.partitioner.PartitionerConfig;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.config.ConfigDef;
public class main{
private static Map<String, String> props = new HashMap<>();
protected static final TopicPartition TOPIC_PARTITION = new TopicPartition(TOPIC, PARTITION);
protected static String url = "hdfs://localhost:9000";
protected static SinkTaskContext context;
public static void main(String[] args) {
HdfsSinkConnector hk = new HdfsSinkConnector();
HdfsSinkTask h = new HdfsSinkTask();
props.put(StorageCommonConfig.STORE_URL_CONFIG, url);
props.put(HdfsSinkConnectorConfig.HDFS_URL_CONFIG, url);
props.put(HdfsSinkConnectorConfig.FLUSH_SIZE_CONFIG, "3");
props.put(HdfsSinkConnectorConfig.FORMAT_CLASS_CONFIG, AvroFormat.class.getName());
try {
hk.start(props);
Collection<SinkRecord> sinkRecords = new ArrayList<>();
SinkRecord record = new SinkRecord("test", 0, null, null, null, null, 0);
sinkRecords.add(record);
h.initialize(context);
h.put(sinkRecords);
hk.stop();
} catch (Exception e) {
throw new ConnectException("Couldn't start HdfsSinkConnector due to configuration error", e);
}
}
}

How to use #Startup Annotation for Service that Pushes Data to Websocket?

This is the Java EE7 #Startup annotation for reference. I am using the latest GlassFish server and IntelliJ to run the server.
I need to instantiate this service so that it can send packets of data periodically to a websocket for processing, but the fact that I use
session.getBasicRemote().sendObject(message);
in the end forces me to throw IOException and EncodeException.
This is bad as #Startup or #Singleton disallows usage of Exceptions when instantiating:
war exploded: java.io.IOException: com.sun.enterprise.admin.remote.RemoteFailureException: Error occurred during deployment: Exception while deploying the app [keyBoard_war_exploded] : The lifecycle method [printSchedule] must not throw a checked exception.
Here is my code:
package site.xxxx.models;
import site.xxxx.message.Message;
import site.xxxx.message.TextMessage;
import site.xxxx.modules.Packet;
import site.xxxx.websocket.MainServerEndpoint;
import javax.annotation.PostConstruct;
import javax.ejb.Schedule;
import javax.ejb.Singleton;
import javax.ejb.Startup;
import javax.json.Json;
import javax.json.JsonObject;
import javax.websocket.EncodeException;
import javax.websocket.Session;
import java.io.IOException;
import java.util.Set;
#Startup
#Singleton
public class Service {
private static Service service;
Set<Session> peers = MainServerEndpoint.peers;
#PostConstruct
public void mainLoop() throws IOException, EncodeException, InterruptedException {
sendSpamPacket();
}
private void sendSpamPacket() throws IOException, EncodeException {
JsonObject ret = Json.createObjectBuilder()
.add("type", "textMessage")
.add("text", "ayyyy")
.build();
Packet packet = new Packet(new TextMessage(ret));
MainServerEndpoint.sendPacket(packet);
//results in calling session.getBasicRemote().sendObject(message);
}
}

Facebook4j authentication issue

I am trying to make a java app which makes connection with facebook.I am using facebok4j to achieve this,I made an app in fb devolopers and got the key annd id for it.But when i am passing it to get an access token its returning an exception error.Please help me
java code
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import net.sf.json.JSONArray;
import net.sf.json.JSONObject;
import facebook4j.Facebook;
import facebook4j.FacebookException;
import facebook4j.FacebookFactory;
import facebook4j.Post;
import facebook4j.ResponseList;
import facebook4j.auth.AccessToken;
import facebook4j.auth.OAuthAuthorization;
import facebook4j.auth.OAuthSupport;
import facebook4j.conf.Configuration;
import facebook4j.conf.ConfigurationBuilder;
public class Fbsample {
public static Configuration createConfiguration()
{
ConfigurationBuilder confBuilder = new ConfigurationBuilder();
confBuilder.setDebugEnabled(true);
confBuilder.setOAuthAppId("*****");
confBuilder.setOAuthAppSecret("*****");
confBuilder.setUseSSL(true);
confBuilder.setJSONStoreEnabled(true);
Configuration configuration = confBuilder.build();
return configuration;
}
public static void main(String[] argv) throws FacebookException {
Configuration configuration = createConfiguration();
FacebookFactory facebookFactory = new FacebookFactory(configuration );
Facebook facebookClient = facebookFactory.getInstance();
AccessToken accessToken = null;
try{
OAuthSupport oAuthSupport = new OAuthAuthorization(configuration );
accessToken = oAuthSupport.getOAuthAppAccessToken();
}catch (FacebookException e) {
System.err.println("Error while creating access token " + e.getLocalizedMessage());
}
facebookClient.setOAuthAccessToken(accessToken);
//results in an error says {An active access token must be used to query information about the current user}
}
}
For now i specified my token and id as *.When running its returning 'Error while creating access token graph.facebook.com'.Thanks in advance.

Lookup datasource with custom variable in runtime

I have context.xml defined in META-INF as follow:
<Context path="/7Restaurant">
<Resource name="datasource"
type="javax.sql.DataSource"
auth="Container"
maxActive="10"
maxIdle="3"
maxWait="10000"
username="work"
password=""
driverClassName="org.postgresql.Driver"
url="jdbc:postgresql://127.0.0.1:5432/7restaurant"/>
</Context>
I am about to deploy my web app (Servlet, in Tomcat 7). The only problem is that, how can I change the url, username, etc. to match that with the environment the WAR package is deployed.
So, in essence, how could such a file is modified; or any other technique so that user can finely connect to the database in the environment it will be deployed on.
I am so new in web-app eclipse, tomcat, postgresql stack; so I expect my question to be wrong, if that's the case; please let me know any other way to solve this.
My context initializer in Java:
package com.restaurant.web;
import java.io.File;
import java.io.InputStream;
import java.io.StringWriter;
import java.net.URL;
import javax.naming.*;
import javax.servlet.*;
import javax.sql.*;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.Result;
import javax.xml.transform.Source;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.*;
import com.restaurant.dao.dbpostgres.DBDAO;
import com.restaurant.setup.GeneralConfigurerSetup;
import com.restaurant.web.Logger;
import sun.java2d.loops.DrawGlyphListAA.General;
public class Database implements ServletContextListener {
private void contextInitialized2(ServletContext servletContext) throws Exception
{
/*
//prepare META-INF/context.xml
DocumentBuilder docbldr = DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document doc = docbldr.newDocument();
Element context = doc.createElement("Context");
context.setAttribute("path", "/7Restaurant");
Element resource = doc.createElement("Resource");
resource.setAttribute("name", "datasource");
resource.setAttribute("type", "javax.sql.DataSource");
resource.setAttribute("auth", "Container");
resource.setAttribute("maxActive", "10");
resource.setAttribute("maxIdle", "3");
resource.setAttribute("maxWait", "10000");
resource.setAttribute("username", Configurer.get(GeneralConfigurerSetup.DB_USERNAME));
resource.setAttribute("password", Configurer.get(GeneralConfigurerSetup.DB_PASSWORD));
resource.setAttribute("driverClassName", Configurer.get(GeneralConfigurerSetup.DB_CLASS));
resource.setAttribute("url", Configurer.get(GeneralConfigurerSetup.DB_JDBCURL));
context.appendChild(resource);
doc.appendChild(context);
System.out.println(System.getProperty("java.class.path"));
File fOut = new File("META-INF/context.xml");
System.out.println(fOut.getAbsolutePath());
if (fOut.exists()) fOut.delete();
TransformerFactory tff = TransformerFactory.newInstance();
Transformer tf = tff.newTransformer();
Source input = new DOMSource(doc);
Result output = new StreamResult(fOut);
tf.transform(input, output);
*/
//META-INF/context
InitialContext enc = new InitialContext();
Context compContext = (Context) enc.lookup("java:comp/env");
DataSource dataSource = (DataSource) compContext.lookup("datasource");
DBDAO.setDataSource(dataSource);
}
public void contextInitialized(ServletContextEvent sce) {
ServletContext servletContext = sce.getServletContext();
try {
contextInitialized2(servletContext);
} catch(Exception e) {
Logger.error("Initializing failed: " + e.getMessage());
System.exit(-1);
}
}
public void contextDestroyed(ServletContextEvent sce) {
}
}

Remove context.xml from your build (just to be safe), and put it in your tomcat folder under conf/Catalina/localhost. Rename it so it's whatever you want your app context name to be (like myapp.xml).
As you can see, that allows each environment to have different myapp.xml files, and when you deploy the war, it will pick that up instead of using an internal one.

embedded ApacheDS in my application, it fails on service.startup() -> DefaultSchemaService.getSchemaManager() returns null

I'm trying to run an embedded ApacheDS in my application. After reading: Running Apache DS embedded in my application
and http://directory.apache.org/apacheds/1.5/41-embedding-apacheds-into-an-application.html
Using the last stable version 1.5.7,
this simple example fails when executing "service.startup();"
Exception in thread "main" java.lang.NullPointerException
at org.apache.directory.server.core.schema.DefaultSchemaService.initialize(DefaultSchemaService.java:380)
at org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1425)
at org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:907)
at Test3.runServer(Test3.java:41)
at Test3.main(Test3.java:24)
that is, DefaultSchemaService.getSchemaManager() returns null.
source code:
import java.util.Properties;
import javax.naming.Context;
import javax.naming.NamingEnumeration;
import javax.naming.NamingException;
import javax.naming.directory.Attributes;
import javax.naming.directory.DirContext;
import javax.naming.directory.InitialDirContext;
import javax.naming.directory.SearchControls;
import javax.naming.directory.SearchResult;
import org.apache.directory.server.core.DefaultDirectoryService;
import org.apache.directory.server.core.partition.Partition;
import org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmPartition;
import org.apache.directory.server.ldap.LdapServer;
import org.apache.directory.server.protocol.shared.transport.TcpTransport;
import org.apache.directory.shared.ldap.entry.ServerEntry;
import org.apache.directory.shared.ldap.name.DN;
public class Test3 {
public static void main(String[] args) throws Exception {
runServer();
testClient();
}
static void runServer() throws Exception {
DefaultDirectoryService service = new DefaultDirectoryService();
service.getChangeLog().setEnabled(false);
Partition partition = new JdbmPartition();
partition.setId("apache");
partition.setSuffix("dc=apache,dc=org");
service.addPartition(partition);
LdapServer ldapService = new LdapServer();
ldapService.setTransports(new TcpTransport(1400));
ldapService.setDirectoryService(service);
service.startup();
// Inject the apache root entry if it does not already exist
try {
service.getAdminSession().lookup(partition.getSuffixDn());
} catch (Exception lnnfe) {
DN dnApache = new DN("dc=Apache,dc=Org");
ServerEntry entryApache = service.newEntry(dnApache);
entryApache.add("objectClass", "top", "domain", "extensibleObject");
entryApache.add("dc", "Apache");
service.getAdminSession().add(entryApache);
}
DN dnApache = new DN("dc=Apache,dc=Org");
ServerEntry entryApache = service.newEntry(dnApache);
entryApache.add("objectClass", "top", "domain", "extensibleObject");
entryApache.add("dc", "Apache");
service.getAdminSession().add(entryApache);
ldapService.start();
}
static void testClient() throws NamingException {
Properties p = new Properties();
p.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
p.setProperty(Context.PROVIDER_URL, "ldap://localhost:1400/");
p.setProperty(Context.SECURITY_PRINCIPAL, "uid=admin,ou=system");
p.setProperty(Context.SECURITY_CREDENTIALS, "secret");
p.setProperty(Context.SECURITY_AUTHENTICATION, "simple");
DirContext rootCtx = new InitialDirContext(p);
DirContext ctx = (DirContext) rootCtx.lookup("dc=apache,dc=org");
SearchControls sc = new SearchControls();
sc.setSearchScope(SearchControls.SUBTREE_SCOPE);
NamingEnumeration<SearchResult> searchResults = ctx.search("", "(objectclass=*)", sc);
while (searchResults.hasMoreElements()) {
SearchResult searchResult = searchResults.next();
Attributes attributes = searchResult.getAttributes();
System.out.println("searchResult.attributes: " + attributes) ;
}
}
}
It seems that ApacheDS versions 1.5.x are not backward compatible.
In ApacheDS 1.5.4 the call "service.startup();" works ok (but it fails somewhere else).
Any idea how to make this example work?

this version of ApacheDS needs to explicitely define the working directory:
service.setWorkingDirectory(new File("data"));

I recently face the similar problem, and after long googling, finally find something useful
Example for Embedded Apache Directory using 1.5.7
http://svn.apache.org/repos/asf/directory/documentation/samples/trunk/embedded-sample/src/main/java/org/apache/directory/seserver/EmbeddedADSVer157.java
and also have a look the pom.xml
Pom.xml Example for Embedded Apache Directory using 1.5.7
http://svn.apache.org/repos/asf/directory/documentation/samples/trunk/embedded-sample/pom.xml
Hope is help!