I'm running Elasticsearch commands from within Java, using Process and ProcessBuilder, on Windows:
new ProcessBuilder(command);
command here is the array of commands:
"C:\\cygwin64\\bin\\curl", "-XGET", "'"+ES_BASE_URL+"index2/_search?pretty'"
The output is fine-- except that the following is prepended to the Cygwin output, i.e. the output i get from Cygwin when i run it directly on a Cygwin terminal:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 796 100 796 0 0 27298 0 --:--:-- --:--:-- --:--:-- 31840
How to avoid this so that i get bare JSon result, the result i get from Cygwin?
curl -s -XGET
should suppress meter. ( You can also reduce it to just a progress bar instead of those numbers by curl -# )
From curl manual:
-#, --progress-bar
Make curl display progress as a simple progress bar instead of the standard, more informational, meter.
-s, --silent
Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for,
potentially
even to the terminal/stdout unless you redirect it.
Related
BACKGROUND INFO:
I am a retired duffer with moderate experience developing windows applications for PC. I am trying to learn java, html, curl or whatever toward learning how to automate an interaction with an online dictionary. My purpose in doing so is to support some word puzzle games I am developing.
My system is a HP laptop running 64-bit Windows Professional. I develop my apps using MS visual studio 2015 Express.
Needless to say, I have no idea what I am doing vis-à-vis curl. Yes, I have read all of the online documentation and tutorials on html, java and curl I could find, but I very quickly find said info over my head.
So please forgive me if this posting is vague or insufficiently specific. I’m doing the best I can.
I feel like I might be on the right track but do not know how to capture the response from the online server. Can someone steer me toward achieving my goal stated above? Thank you for attending to this.
Robert Hoech
ISSUE STATEMENT:
CMD:
C:\Users\Robert\Documents\Rob\CURL>curl -v http://www.merriam-webster/dictionary
/capricious >data.txt
WRITTEN TO data.txt
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
WRITTEN TO CONSOLE:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0*
Trying 198.105.254.23:80...
0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0*
connect to 198.105.254.23 port 80 failed: Timed out
* Trying 198.105.244.23:80...
0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0*
Connected to www.merriam-webster (198.105.244.23) port 80 (#0)
> GET /dictionary/capricious HTTP/1.1
> Host: www.merriam-webster
> User-Agent: curl/7.71.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Fri, 24 Jul 2020 20:48:42 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: close
< Location: http://localhost
< Expires: Fri, 24 Jul 2020 20:48:41 GMT
< Cache-Control: no-cache
<
{ [189 bytes data]
100 178 0 178 0 0 8 0 --:--:-- 0:00:21 --:--:-- 43
* Closing connection 0
The response you are getting via "curl" is a 301 Redirect to "http://localhost". That means read the file off "your machine". Obviously, that won't work ...
I think that this is the Merriam-Webster site telling you that what you are trying to do is not permitted by their Terms of Service.
If you wanted to ignore that (and risk a possible lawsuit!) you could try changing the agent string to trick the website into thinking your application is a web browser.
A better idea would be:
Contact Merriam-Webster to see if there is an API service (or something) that you are permit you to use. (You may have to pay for it.)
Trying and find an alternative free online dictionary. (Make sure that you read the Terms of Service thoroughly!)
Find a free fixed word list and embed it in your application.
UPDATE - So I tried this myself, and here is what I found:
The URL you are using is incorrect:
http://www.merriam-webster/dictionary/capricious
The hostname part is incorrect. It should be "www.merriam-webster.com".
If you use "http:" with the correct hostname, it redirects to "https:"
Curling
https://www.merriam-webster.com/dictionary/capricious
gives an HTML document which looks like it has the definition in it. Whether it is scrape-able in practice ... I can't say. (But I couldn't get the page to pay attention to an "Accept: text/plain" header.)
But I also found that there is an official Merriam-Webster API; see https://dictionaryapi.com/. The page says that there is a free / non-commercial use option.
I have spring boot application with embedded jetty and its configurations are:
jetty's minThread: 50
jetty's maxThread: 500
jetty's maxQueueSize: 25000 (I changed default queue to LinkedBlockingQueue)
I didn't change acceptors and selectors (since I dont believe on hard coding the value)
With above configuration, I am getting below jmeter test results:
Concurrent Users: 60
summary = 183571 in 00:01:54 = 1611.9/s Avg: 36 Min: 3 Max:
1062 Err: 0 (0.00%)
Concurrent Users: 75
summary = 496619 in 00:05:00 = 1654.6/s Avg: 45 Min: 3 Max:
1169 Err: 0 (0.00%)
If I increase concurrent users, I dont see any improvement. I want to increase concurrency. How to achieve this?
===========================================================================
Updating on 29-March-2019
I was spending more effort on improving business logic. Still no much improvement. Then I decided to develop one hello world spring-boot project.
i.e.,
spring-boot (1.5.9)
jetty 9.4.15
rest controller which has get endpoint
code below:
#GetMapping
public String index() {
return "Greetings from Spring Boot!";
}
Then I tried to benchmark using apachebench
75 concurrent users:
ab -t 120 -n 1000000 -c 75 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 75
Time taken for tests: 37.184 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 26893.28 [#/sec] (mean)
Time per request: 2.789 [ms] (mean)
Time per request: 0.037 [ms] (mean, across all concurrent requests)
Transfer rate: 3755.61 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 23.5 0 3006
Processing: 0 2 7.8 1 404
Waiting: 0 2 7.8 1 404
Total: 0 3 24.9 2 3007
100 concurrent users:
ab -t 120 -n 1000000 -c 100 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 100
Time taken for tests: 36.708 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 27241.77 [#/sec] (mean)
Time per request: 3.671 [ms] (mean)
Time per request: 0.037 [ms] (mean, across all concurrent requests)
Transfer rate: 3804.27 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 35.7 1 3007
Processing: 0 2 9.4 1 405
Waiting: 0 2 9.4 1 405
Total: 0 4 37.0 2 3009
500 concurrent users:
ab -t 120 -n 1000000 -c 500 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 500
Time taken for tests: 36.222 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 27607.83 [#/sec] (mean)
Time per request: 18.111 [ms] (mean)
Time per request: 0.036 [ms] (mean, across all concurrent requests)
Transfer rate: 3855.39 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 14 126.2 1 7015
Processing: 0 4 22.3 1 811
Waiting: 0 3 22.3 1 810
Total: 0 18 129.2 2 7018
1000 concurrent users:
ab -t 120 -n 1000000 -c 1000 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 1000
Time taken for tests: 36.534 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 27372.09 [#/sec] (mean)
Time per request: 36.534 [ms] (mean)
Time per request: 0.037 [ms] (mean, across all concurrent requests)
Transfer rate: 3822.47 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 30 190.8 1 7015
Processing: 0 6 31.4 2 1613
Waiting: 0 5 31.4 1 1613
Total: 0 36 195.5 2 7018
From above test run, I achieved ~27K per second with 75 users itself but it looks increasing the users also increasing the latency. Also, we can clearly note connect time is increasing.
I have requirement for my application to support 40k concurrent users (assume all are using own separate browsers) and request should be finished within 250 milliseconds.
Please help me on this
You can try increasing or decreasing the number of Jetty threads but the application performance will depend on the application logic. If your current bottleneck is the database query you will see hardly any improvements by tuning HTTP layer, especially when testing over local network.
Find the bottleneck in your application, attempt to improve it, and then measure again to confirm it's better. Repeat this three steps until achieving desired performance. Do not tune performance blindly, it's a waste of time.
This question already has answers here:
is nice() used to change the thread priority or the process priority?
(3 answers)
Closed 1 year ago.
On a Unix system, you can run a process at lower CPU "priority" (pedantically, it does not change the thing that is called the priority, but rather influences what share of available CPU time is used, which is "priority" in the general sense) using the nice command:
nice program
And you could use that to run a JVM process:
nice java -jar program.jar
The Java program run by that JVM process will start multiple threads.
Does the nice change affect the scheduling of those Java threads? That is, will the Java threads have a lower CPU priority when run as
nice java -jar program.jar
that when run as
java -jar program.jar
In general, this will be system dependent, so I am interested in the Linux case.
According to what ps reports niceness is applied to java threads. I ran this quick test with a java application that waits for user input:
Start process with : nice -n 19 java Main
Output of ps -m -l 20746
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
0 - 1000 20746 10006 0 - - - 1739135 - pts/2 0:00 java Main
0 S 1000 - - 0 99 19 - - futex_ - 0:00 -
1 S 1000 - - 0 99 19 - - wait_w - 0:00 -
1 S 1000 - - 0 99 19 - - futex_ - 0:00 -
1 S 1000 - - 0 99 19 - - futex_ - 0:00 -
Start process with : nice -n 15 java Main
Output of ps -m -l 21488
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
0 - 1000 21488 10006 0 - - - 1722494 - pts/2 0:00 java Main
0 S 1000 - - 0 95 15 - - futex_ - 0:00 -
1 S 1000 - - 0 95 15 - - wait_w - 0:00 -
1 S 1000 - - 0 95 15 - - futex_ - 0:00 -
1 S 1000 - - 0 95 15 - - futex_ - 0:00 -
The NI column seems to reflect what I passed to nice and the priority changes accordingly too. I got the process ID (20746, 21488) using jps.
Note that running jstack 21488 for example will not give the above information.
I ran the above on Ubuntu 16.04 LTS (64bit)
Actually...Niceness is a property of the application according to POSIX.1. Here is a more detailed post. is nice() used to change the thread priority or the process priority?
Java is not special. It's just a process, and the OS sets its "niceness" the same way as with any other process.
On Linux, Java threads are implemented using native threads, so again, "niceness" is subject to OS controls in the same way as any other native thread.
I have an akka (Java) application with camel-jetty consumer. Under some minimum load (about 10 TPS), our client starts seeing HTTP 503 error. I tried to reproduce the problem in our lab, and it seems jetty can't handle overlapping HTTP requests. Below is the output from apache bench (ab):
ab sends 10 requests using one single thread (i.e. one request at a time)
ab -n 10 -c 1 -p bad.txt http://192.168.20.103:8899/pim
Benchmarking 192.168.20.103 (be patient).....done
Server Software: Jetty(8.1.16.v20140903)
Server Hostname: 192.168.20.103
Server Port: 8899
Document Path: /pim
Document Length: 33 bytes
Concurrency Level: 1
Time taken for tests: 0.61265 seconds
Complete requests: 10
Failed requests: 0
Requests per second: 163.23 [#/sec] (mean)
Time per request: 6.126 [ms] (mean)
Time per request: 6.126 [ms] (mean, across all concurrent requests)
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 1 2
Processing: 3 4 1.8 5 7
Waiting: 2 4 1.8 5 7
Total: 3 5 1.9 6 8
Percentage of the requests served within a certain time (ms)
50% 6
66% 6
75% 6
80% 8
90% 8
95% 8
98% 8
99% 8
100% 8 (longest request)
ab sends 10 requests using two threads (up to 2 requests at the same time):
ab -n 10 -c 2 -p bad.txt http://192.168.20.103:8899/pim
Benchmarking 192.168.20.103 (be patient).....done
Server Software: Jetty(8.1.16.v20140903)
Server Hostname: 192.168.20.103
Server Port: 8899
Document Path: /pim
Document Length: 33 bytes
Concurrency Level: 2
Time taken for tests: 30.24549 seconds
Complete requests: 10
Failed requests: 1
(Connect: 0, Length: 1, Exceptions: 0)
// obmited for clarity
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.9 1 2
Processing: 3 3005 9492.9 4 30023
Waiting: 2 3005 9492.7 3 30022
Total: 3 3006 9493.0 5 30024
Percentage of the requests served within a certain time (ms)
50% 5
66% 5
75% 7
80% 7
90% 30024
95% 30024
98% 30024
99% 30024
100% 30024 (longest request)
I don't believe jetty is this bad. Hopefully, it's just a configuration issue. This is the setting for my camel consumer URI:
"jetty:http://0.0.0.0:8899/pim?replyTimeout=70000&autoAck=false"
I am using akka 2.3.12 and camel-jetty 2.15.2
Jetty is certain not that bad and should be able to handle 10s of thousands of connections with many thousands of TPS.
Hard to diagnose from what you have said, other than Jetty does not send 503's when it is under load.... unless perhaps if the Denial of Service protection filter is deployed? (and ab would look like a DOS attack.... which it basically is and is not a great load generator for benchmarking).
So you need to track down who/what is sending that 503 and why.
It was my bad code: the sender (client) info was overwritten with overlapping requests. The 503 error message was sent due to Jetty continuation timeout.
I want to detect the Memory and CPU consumption of a particular app in android (programmatically), can any one help me with it. I have tried the TOP method, but i want an alternative for it.
Any help will be appreciated, thanks :)
If you wan to trace your memory usage in your app then there is ActivityManager.getMemoryInfo() API.
Cpu usage can be traced using CpuStatsCollector API.
For more informative memory usage overview, outside your app, you can use adb shell dumpsys meminfo <package_name|pid> [-d] for more specific memory usage statistics. For example, there is the the command for com.google.android.apps.maps process:
adb shell dumpsys meminfo com.google.android.apps.maps -d
Which gives you a following output:
** MEMINFO in pid 18227 [com.google.android.apps.maps] **
Pss Private Private Swapped Heap Heap Heap
Total Dirty Clean Dirty Size Alloc Free
------ ------ ------ ------ ------ ------ ------
Native Heap 10468 10408 0 0 20480 14462 6017
Dalvik Heap 34340 33816 0 0 62436 53883 8553
Dalvik Other 972 972 0 0
Stack 1144 1144 0 0
Gfx dev 35300 35300 0 0
Other dev 5 0 4 0
.so mmap 1943 504 188 0
.apk mmap 598 0 136 0
.ttf mmap 134 0 68 0
.dex mmap 3908 0 3904 0
.oat mmap 1344 0 56 0
.art mmap 2037 1784 28 0
Other mmap 30 4 0 0
EGL mtrack 73072 73072 0 0
GL mtrack 51044 51044 0 0
Unknown 185 184 0 0
TOTAL 216524 208232 4384 0 82916 68345 14570
(output trimmed) More about it here
Tracing of memory usage on modern operating systems is very complex task. See this question for more info.
To get your processid:
int pid = android.os.Process.myPid();
To get CPU Usage :
public String getCPUUsage(int pid) {
Process p;
try {
String[] cmd = {
"sh",
"-c",
"top -m 1000 -d 1 -n 1 | grep \""+pid+"\" "};
p = Runtime.getRuntime().exec(cmd);
String line = reader.readLine();
// line contains the process info
}