IN order to test couchbase, I am trying to create 30K-specific documents in 15 minutes.
During the test, 6563 documents are created and then hangs. I have seen that it takes 2 minutes to create 0-3K thousand; 5 minutes to create between 3K-6K; and 5 minutes to create the final 6K -6.5K documents.
Example here.
I would appreciate help in understanding what I am doing wrong. The code is below:
public class ConnectionManager {
Logger logger = Logger.getLogger(getClass().getName());
private CouchbaseClient client;
public ConnectionManager() {
init();
}
public void init() {
try {
logger.info("Opening base connection.");
List<URI> hosts = Arrays.asList(new URI("http://127.0.0.1:8091/pools"));
String bucket = "default";
String password = "";
client = new CouchbaseClient(hosts, bucket, password);
} catch (Exception e) {
client = null;
throw new IllegalStateException(e);
}
}
#PreDestroy
public void destroy() {
logger.info("Closing base connection.");
if (client != null) {
client.shutdown();
client = null;
}
}
public CouchbaseClient getClient() {
return client;
}
}
public class DatabaseManager {
ConnectionManager cm;
public DatabaseManager() {
cm = new ConnectionManager();
}
public String addDocument(String result) {
CouchbaseClient c = cm.getClient();
JSONParameters j = new JSONParameters();
String id = UUID.randomUUID().toString();
Date today = new Date();
SimpleDateFormat DATE_FORMAT = new SimpleDateFormat("yyyy.MM.dd HH:mm:ss.SSSZ");
String date = DATE_FORMAT.format(today);
j.setTime(date);
j.setData(UUID.randomUUID().toString());
j.setSender_id(result);
j.setFlag(false);
Gson gson = new Gson();
String json = gson.toJson(j);
c.add(result, json);
return json;
}
public class DataBaseAddServlet extends HttpServlet {
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
try {
for (int k = 0; k < 30000; k++) {
String id = UUID.randomUUID().toString();
DatabaseManager dbManager = new DatabaseManager();
dbManager.addDocument(id);
}
} catch (Exception e) {
resp.getOutputStream().println(e.getMessage());
resp.flushBuffer();
}
}
}
I think you missed a key point from the example you linked to: in the Servlet, DatabaseManager is injected via dependency injection and thus there's only one instance created.
In you code, you actually create a new DatabaseManager inside the loop, so you end up creating 30K CouchbaseClients. You are probably hitting a limit, and definitely wasting a lot of time and resources with that much extra clients.
Just moving the DatabaseManager dbManager = new DatabaseManager(); before the for loop should make things far better.
Related
I run a custom WebSocketServlet for Jetty, which sends short text push notifications (for an async mobile and desktop word game) to many platforms (Facebook, Vk.com, Mail.ru, Ok.ru also Firebase and Amazon messaging) using a Jetty HttpClient instance:
public class MyServlet extends WebSocketServlet {
private final SslContextFactory mSslFactory = new SslContextFactory();
private final HttpClient mHttpClient = new HttpClient(mSslFactory);
#Override
public void init() throws ServletException {
super.init();
try {
mHttpClient.start();
} catch (Exception ex) {
throw new ServletException(ex);
}
mFcm = new Fcm(mHttpClient); // Firebase
mAdm = new Adm(mHttpClient); // Amazon
mApns = new Apns(mHttpClient); // Apple
mFacebook = new Facebook(mHttpClient);
mMailru = new Mailru(mHttpClient);
mOk = new Ok(mHttpClient);
mVk = new Vk(mHttpClient);
}
This has worked very good for the past year, but since I have recently upgraded my WAR-file to use Jetty 9.4.14.v20181114 the trouble has begun -
public class Facebook {
private final static String APP_ID = "XXXXX";
private final static String APP_SECRET = "XXXXX";
private final static String MESSAGE_URL = "https://graph.facebook.com/%s/notifications?" +
// the app access token is: "app id | app secret"
"access_token=%s%%7C%s" +
"&template=%s";
private final HttpClient mHttpClient;
public Facebook(HttpClient httpClient) {
mHttpClient = httpClient;
}
private final BufferingResponseListener mMessageListener = new BufferingResponseListener() {
#Override
public void onComplete(Result result) {
if (!result.isSucceeded()) {
LOG.warn("facebook failure: {}", result.getFailure());
return;
}
try {
// THE jsonStr SUDDENLY CONTAINS PREVIOUS CONTENT!
String jsonStr = getContentAsString(StandardCharsets.UTF_8);
LOG.info("facebook success: {}", jsonStr);
} catch (Exception ex) {
LOG.warn("facebook exception: ", ex);
}
}
};
public void postMessage(int uid, String sid, String body) {
String url = String.format(MESSAGE_URL, sid, APP_ID, APP_SECRET, UrlEncoded.encodeString(body));
mHttpClient.POST(url).send(mMessageListener);
}
}
Suddenly the getContentAsString method called for successful HttpClient invocations started to deliver the strings, which were fetched previously - prepended to the the actual result string.
What could it be please, is it some changed BufferingResponseListener behaviour or maybe some non-obvious Java quirk?
BufferingResponseListener was never intended to be reusable across requests.
Just allocate a new BufferingResponseListener for every request/response.
I have to write some dao tests for project where I want to:
create DDL schema from database (MySQL);
create tables in another test database in memory (H2);
insеrt some data to database;
select the just inserted item;
check some data from this item.
This is my test:
public class BridgeDBTest {
private static String JDBC_DRIVER;
private static String JDBC_URL;
private static String USER;
private static String PSWD;
private static final Logger logger = LoggerFactory.getLogger(BridgeDBTest.class);
#BeforeGroups(groups = "bridgeDB")
public void init(){
try {
JDBC_DRIVER = org.h2.Driver.class.getName();
JDBC_URL = "jdbc:h2:mem:test;DB_CLOSE_DELAY=-1";
USER = "root";
PSWD = "";
new HibernateTestUtil().setDialect("org.hibernate.dialect.HSQLDialect")
.translateCreateDllToOutputStream(new FileOutputStream(new File("src/test/resources/createSchema.sql")));
RunScript.execute(JDBC_URL, USER, PSWD, "src/test/resources/createSchema.sql", Charset.forName("UTF8"), false);
insertDataset(readDataSet());
}
catch (Exception expt) {
expt.printStackTrace();
logger.error("!!!" + expt);
throw new RuntimeException(expt.getMessage());
}
}
#Test(groups = "bridgeDB")
public void getItem(){
BridgeDAOImpl dao = new BridgeDAOImpl();
dao.setSessionFactory(new HibernateTestUtil().getSessionFactory());
try {
Bridge bridge = dao.get(1L);
assert(bridge.getName().equals("TEST-CN-DEVBOX01"));
} catch (ServiceException e) {
e.printStackTrace();
}
}
#AfterGroups(groups = "bridgeDB")
public void dropTables(){
try {
new HibernateTestUtil().setDialect("org.hibernate.dialect.HSQLDialect")
.translateDropDllToOutputStream(new FileOutputStream(new File("src/test/resources/dropSchema.sql")));
}
catch (Exception expt) {
expt.printStackTrace();
logger.error("!!!" + expt);
throw new RuntimeException(expt.getMessage());
}
}
private IDataSet readDataSet() throws Exception{
return new FlatXmlDataSetBuilder().build(new File("src/test/resources/datasetForTest.xml"));
}
private void insertDataset(IDataSet dataSet) throws Exception{
IDatabaseTester databaseTester = new JdbcDatabaseTester(JDBC_DRIVER, JDBC_URL, USER, PSWD);
databaseTester.setSetUpOperation(DatabaseOperation.CLEAN_INSERT);
databaseTester.setDataSet(dataSet);
databaseTester.onSetup();
}
}
BridgeDAOImplused class HibernateUtilfrom src/main/..., but I need to use my class HibernateTestUtil from src/test/.... It's modified HibernateUtil fitted for my test (there I set parameters for Configuration class).
BridgeDAOImpl (See 5 line in try block):
public class BridgeDAOImpl extends GenericDAOImpl<Bridge, Long> implements BridgeDAO {
//...
public SearchResult<Bridge> list(int from, int limit, String filter, String order, Long authId) throws ServiceException {
SearchResult<Bridge> results = null;
Search search = new Search(Bridge.class);
Session session = getSessionFactory().getCurrentSession();
Transaction transaction = null;
try {
transaction = session.beginTransaction();
search.setFirstResult(from);
search.setMaxResults(limit);
HibernateUtil.buildSearch(filter, order, search, aliases);
results = searchAndCount(search);
transaction.commit();
}
catch (Exception expt) {
logger.error("!!!", expt);
if (transaction != null) {
transaction.rollback();
}
throw new ServiceException(expt.getMessage());
}
finally {
// session.close();
}
return results;
}
//...
}
How I can test my dao without modifying it?
I have some old playframework 2.2 java webservice that interacts with akka, and now I should port them to playframework 2.3.
However, async has been deprecated and even after reading the doc about the async porting (http://www.playframework.com/documentation/2.3.x/JavaAsync) I wasn't able to understand how to apply it to my case (code below):
I must make the await for a timeout/akka server reply before starting the construction of my reply (ok()), otherwise I will block the thread.
I should make the actorselection async too.
I should make the akka server reply parsing/reply construction async too
I looked around and I wasn't able to find an example of such interactions, even in typesafe templates.
How could I do that?
/* playframework 2.2 code */
public class Resolve extends Controller {
private final static String RESOLVER_ACTOR = play.Play.application().configuration().getString("actor.resolve");
#CorsRest
#VerboseRest
#RequireAuthentication
#BodyParser.Of(BodyParser.Json.class)
public static Result getJsonTree() {
JsonNode json = request().body().asJson();
ProtoBufMessages.ResolveRequest msg;
ResolveRequestInput input;
try {
input = new ResolveRequestInput(json);
} catch (rest.exceptions.MalformedInputException mie) {
return badRequest(mie.getMessage());
}
msg = ((ProtoBufMessages.ResolveRequest)input.getMessage());
ActorSelection resolver = Akka.system().actorSelection(RESOLVER_ACTOR);
Timeout tim = new Timeout(Duration.create(4, "seconds"));
Future<Object> fut = Patterns.ask(resolver, input.getMessage(), tim);
return async (
F.Promise.wrap(fut).map(
new F.Function<Object, Result>() {
public Result apply(Object response) {
ProtoBufMessages.ResolveReply rsp = ((ProtoBufMessages.ResolveReply)response);
ResolveOutput output = new ResolveOutput(rsp);
return ok(output.getJsonReply());
}
}
)
);
}
}
I came out with the code below
public class Resolve extends Controller {
private final static String RESOLVER_ACTOR = play.Play.application().configuration().getString("actor.resolve");
private final static BrainProtoMessages.ResolveReply request_error = BrainProtoMessages.ResolveReply.newBuilder()
.setReturnCode(BResults.REQUEST_FAILED)
.build();
#CorsRest
#VerboseRest
#RequireAuthentication
#BodyParser.Of(BodyParser.Json.class)
public static Result resolve_map() {
final ResolveRequestInput input;
final F.Promise<ActorSelection> selected_target;
final F.Promise<Future<Object>> backend_request;
final F.Promise<BrainProtoMessages.ResolveReply> backend_reply;
final F.Promise<ObjectNode> decode_json;
final F.Promise<Result> ok_result;
final JsonNode json = request().body().asJson();
try {
input = new ResolveRequestInput(json);
} catch (rest.exceptions.MalformedInputException mie) {
return badRequest(mie.getMessage());
}
selected_target = F.Promise.promise(
new F.Function0<ActorSelection>() {
#Override
public ActorSelection apply() throws Throwable {
return Akka.system().actorSelection(RESOLVER_ACTOR);
}
}
);
backend_request =
selected_target.map(
new F.Function<ActorSelection, Future<Object>>() {
#Override
public Future<Object> apply(ActorSelection actorSelection) throws Throwable {
return Patterns.ask(actorSelection, input.getMessage(),new Timeout(Duration.create(4, "seconds")));
}
}
);
backend_reply = backend_request.map(
new F.Function<Future<Object>, BrainProtoMessages.ResolveReply>() {
#Override
public BrainProtoMessages.ResolveReply apply(Future<Object> akka_reply) throws Throwable {
try {
return (BrainProtoMessages.ResolveReply) Await.result(akka_reply, Duration.create(4, "seconds"));
}catch(Exception error)
{
return request_error;
}
}
}
);
decode_json = backend_reply.map(
new F.Function<BrainProtoMessages.ResolveReply, ObjectNode>() {
#Override
public ObjectNode apply(BrainProtoMessages.ResolveReply response) throws Throwable {
return new ResolveOutput(response).getJsonReply();
}
}
);
ok_result = decode_json.map(
new F.Function<ObjectNode, Result>() {
#Override
public Result apply(ObjectNode reply) {
return ok(reply);
}
}
);
try {
return ok_result.get(8000);
}catch(Exception error)
{
return internalServerError();
}
}
}
when I try to request a resource from my API via 100 threads at the same time (It's a simple performance test) I get the following server error message:
Unable to run the following server-side task: Handle inbound messages
I already searched the internet but couldn't find any hint how to get rid of it.
Any experience how to handle that error?
Greetingz,
Cooks
Edit:
Client:
public class PerformanceTest extends Thread {
static ClientResource client;
static int recCounter = 0;
public PerformanceTest(){
super();
}
public static void main(String[] args){
client = new ClientResource("http://localhost:8082/api/module/news/1");
ChallengeScheme scheme = ChallengeScheme.HTTP_BASIC;
ChallengeResponse authentication = new ChallengeResponse(scheme, "app+postapp", "");
client.setChallengeResponse(authentication);
for(int i = 0; i < 100; i++){
System.out.println("Starting test: " + i);
new PerformanceTest().start();
}
}
#Override
public void run() {
JsonRepresentation entity;
try {
entity = new JsonRepresentation(client.get());
recCounter++;
System.out.println("(" + getRecCounter() + ") result: " + entity.getJsonObject().toString());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private synchronized int getRecCounter(){
return recCounter;
}
Server:
This is my Application Class that starts the server:
public class MyApplication extends Application {
private DB db;
public MyApplication() {
db = new DB();
}
public static void main(String[] args) throws Exception {
Component component = new Component();
component.getServers().add(Protocol.HTTP, 8082);
Application application = new MyApplication();
// Attach the application to the component with a defined contextroot
String contextRoot = "/api";
component.getDefaultHost().attach(contextRoot, application);
component.start();
}
public Restlet createInboundRoot() {
router.attach("/module/news", NewsResource.class);
router.attach("/module/news/{itemId}", NewsResource.class);
router.attach("/module/news/item", NewsResource.class);
router.attach("/module/news/item/{itemId}", NewsResource.class);
// Create Verifier
CredentialsVerifier verifier = new CredentialsVerifier(getDB());
// Create a Guard
ChallengeAuthenticator guard = new ChallengeAuthenticator(getContext(), ChallengeScheme.HTTP_BASIC, "app");
guard.setVerifier(verifier);
guard.setNext(router);
return guard;
}
public synchronized DB getDB() {
return this.db;
}
}
Does anyone know where to find a little how to on using dbpedia spotlight in java or scala? Or could anyone explain how it's done? I can't find any information on this...
The DBpedia Spotlight wiki pages would be a good place to start.
And I believe the installation page has listed the most popular ways (using a jar, or set up a web service) to use the application.
It includes instructions on using the Java/Scala API with your own installation, or calling the Web Service.
There are some additional data needed to be downloaded to run your own server for full service, good time to make a coffee for yourself.
you need download dbpedia spotlight (jar file) after that u can use next two classes ( author pablomendes ) i only make some change .
public class db extends AnnotationClient {
//private final static String API_URL = "http://jodaiber.dyndns.org:2222/";
private static String API_URL = "http://spotlight.dbpedia.org:80/";
private static double CONFIDENCE = 0.0;
private static int SUPPORT = 0;
private static String powered_by ="non";
private static String spotter ="CoOccurrenceBasedSelector";//"LingPipeSpotter"=Annotate all spots
//AtLeastOneNounSelector"=No verbs and adjs.
//"CoOccurrenceBasedSelector" =No 'common words'
//"NESpotter"=Only Per.,Org.,Loc.
private static String disambiguator ="Default";//Default ;Occurrences=Occurrence-centric;Document=Document-centric
private static String showScores ="yes";
#SuppressWarnings("static-access")
public void configiration(double CONFIDENCE,int SUPPORT,
String powered_by,String spotter,String disambiguator,String showScores){
this.CONFIDENCE=CONFIDENCE;
this.SUPPORT=SUPPORT;
this.powered_by=powered_by;
this.spotter=spotter;
this.disambiguator=disambiguator;
this.showScores=showScores;
}
public List<DBpediaResource> extract(Text text) throws AnnotationException {
LOG.info("Querying API.");
String spotlightResponse;
try {
String Query=API_URL + "rest/annotate/?" +
"confidence=" + CONFIDENCE
+ "&support=" + SUPPORT
+ "&spotter=" + spotter
+ "&disambiguator=" + disambiguator
+ "&showScores=" + showScores
+ "&powered_by=" + powered_by
+ "&text=" + URLEncoder.encode(text.text(), "utf-8");
LOG.info(Query);
GetMethod getMethod = new GetMethod(Query);
getMethod.addRequestHeader(new Header("Accept", "application/json"));
spotlightResponse = request(getMethod);
} catch (UnsupportedEncodingException e) {
throw new AnnotationException("Could not encode text.", e);
}
assert spotlightResponse != null;
JSONObject resultJSON = null;
JSONArray entities = null;
try {
resultJSON = new JSONObject(spotlightResponse);
entities = resultJSON.getJSONArray("Resources");
} catch (JSONException e) {
//throw new AnnotationException("Received invalid response from DBpedia Spotlight API.");
}
LinkedList<DBpediaResource> resources = new LinkedList<DBpediaResource>();
if(entities!=null)
for(int i = 0; i < entities.length(); i++) {
try {
JSONObject entity = entities.getJSONObject(i);
resources.add(
new DBpediaResource(entity.getString("#URI"),
Integer.parseInt(entity.getString("#support"))));
} catch (JSONException e) {
LOG.error("JSON exception "+e);
}
}
return resources;
}
}
second class
/**
* #author pablomendes
*/
public abstract class AnnotationClient {
public Logger LOG = Logger.getLogger(this.getClass());
private List<String> RES = new ArrayList<String>();
// Create an instance of HttpClient.
private static HttpClient client = new HttpClient();
public List<String> getResu(){
return RES;
}
public String request(HttpMethod method) throws AnnotationException {
String response = null;
// Provide custom retry handler is necessary
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
try {
// Execute the method.
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
LOG.error("Method failed: " + method.getStatusLine());
}
// Read the response body.
byte[] responseBody = method.getResponseBody(); //TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
response = new String(responseBody);
} catch (HttpException e) {
LOG.error("Fatal protocol violation: " + e.getMessage());
throw new AnnotationException("Protocol error executing HTTP request.",e);
} catch (IOException e) {
LOG.error("Fatal transport error: " + e.getMessage());
LOG.error(method.getQueryString());
throw new AnnotationException("Transport error executing HTTP request.",e);
} finally {
// Release the connection.
method.releaseConnection();
}
return response;
}
protected static String readFileAsString(String filePath) throws java.io.IOException{
return readFileAsString(new File(filePath));
}
protected static String readFileAsString(File file) throws IOException {
byte[] buffer = new byte[(int) file.length()];
#SuppressWarnings("resource")
BufferedInputStream f = new BufferedInputStream(new FileInputStream(file));
f.read(buffer);
return new String(buffer);
}
static abstract class LineParser {
public abstract String parse(String s) throws ParseException;
static class ManualDatasetLineParser extends LineParser {
public String parse(String s) throws ParseException {
return s.trim();
}
}
static class OccTSVLineParser extends LineParser {
public String parse(String s) throws ParseException {
String result = s;
try {
result = s.trim().split("\t")[3];
} catch (ArrayIndexOutOfBoundsException e) {
throw new ParseException(e.getMessage(), 3);
}
return result;
}
}
}
public void saveExtractedEntitiesSet(String Question, LineParser parser, int restartFrom) throws Exception {
String text = Question;
int i=0;
//int correct =0 ; int error = 0;int sum = 0;
for (String snippet: text.split("\n")) {
String s = parser.parse(snippet);
if (s!= null && !s.equals("")) {
i++;
if (i<restartFrom) continue;
List<DBpediaResource> entities = new ArrayList<DBpediaResource>();
try {
entities = extract(new Text(snippet.replaceAll("\\s+"," ")));
System.out.println(entities.get(0).getFullUri());
} catch (AnnotationException e) {
// error++;
LOG.error(e);
e.printStackTrace();
}
for (DBpediaResource e: entities) {
RES.add(e.uri());
}
}
}
}
public abstract List<DBpediaResource> extract(Text text) throws AnnotationException;
public void evaluate(String Question) throws Exception {
evaluateManual(Question,0);
}
public void evaluateManual(String Question, int restartFrom) throws Exception {
saveExtractedEntitiesSet(Question,new LineParser.ManualDatasetLineParser(), restartFrom);
}
}
main()
public static void main(String[] args) throws Exception {
String Question ="Is the Amazon river longer than the Nile River?";
db c = new db ();
c.configiration(0.0, 0, "non", "CoOccurrenceBasedSelector", "Default", "yes");
System.out.println("resource : "+c.getResu());
}
I just add one little fix for your answer.
Your code is running, if you add the evaluate method call:
public static void main(String[] args) throws Exception {
String question = "Is the Amazon river longer than the Nile River?";
db c = new db ();
c.configiration(0.0, 0, "non", "CoOccurrenceBasedSelector", "Default", "yes");
c.evaluate(question);
System.out.println("resource : "+c.getResu());
}
Lamine
In the request method of the second class (AnnotationClient) in Adel's answer, the author Pablo Mendes hasn't finished
TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
which is an annoying warning that needs to be removed by replacing
byte[] responseBody = method.getResponseBody(); //TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
response = new String(responseBody);
with
Reader in = new InputStreamReader(method.getResponseBodyAsStream(), "UTF-8");
StringWriter writer = new StringWriter();
org.apache.commons.io.IOUtils.copy(in, writer);
response = writer.toString();