I'm working with Nifi and I'm trying to create an Apache Nifi processor but it doesn't react ! The processor is built, it shows up in the processors list and it works but without any inputs !! it does not show me anything.
Here is my code :
#Override
public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
final AtomicReference<String> value = new AtomicReference<>();
Integer Somme=new Integer(1254+5520);
FlowFile flowFile = session.get();
session.read(flowFile, new InputStreamCallback() {
#Override
public void process(InputStream in) throws IOException {
HttpGet request = new HttpGet(URL);
HttpHost target = new HttpHost(localhost, 8080, "http");
CredentialsProvider provider = new BasicCredentialsProvider();
provider.setCredentials(
new AuthScope(target.getHostName(), target.getPort()),
new UsernamePasswordCredentials(Username, paswd)
);
AuthCache authCache = new BasicAuthCache();
authCache.put(target, new BasicScheme());
HttpClientContext localContext = HttpClientContext.create();
localContext.setAuthCache(authCache);
try (CloseableHttpClient httpClient = HttpClientBuilder.create()
.setDefaultCredentialsProvider(provider)
.build();
CloseableHttpResponse response = httpClient.execute(target, request, localContext)) {
// 401 if wrong user/password
System.out.println(response.getStatusLine().getStatusCode());
HttpEntity entity = response.getEntity();
if (entity != null) {
// return it as a String
String result = EntityUtils.toString(entity);
System.out.println(result.getBytes());
value.set(result);
}
}
}
});
// Write the results to an attribute
String results = value.get();
if(results != null && !results.isEmpty()){
flowFile = session.putAttribute(flowFile, "match", results);
}
// To write the results back out ot flow file
flowFile = session.write(flowFile, new OutputStreamCallback() {
#Override
public void process(OutputStream out) throws IOException {
out.write(value.get().getBytes());
}
});
session.transfer(flowFile, Success);
}
}
Any help please ! It's been days and nothing seems right :(
When implementing a source processor (meaning there is no input), then you want to use session.create() to create a brand new flow file. Using session.get() will attempt to get a flow file from incoming queues, but since there are none, it will always be null.
What is the easiest way to give HttpClient a list of routes, host names, IP addresses that should be prevented from access?
Something like this (this code doesn't work, you can't set 0 now):
HttpHost block = new HttpHost("example.com", 80);
cm.setMaxPerRoute(new HttpRoute(block), 0);
final Set<HttpHost> sitesToBlock = new HashSet<HttpHost>();
sitesToBlock.add(new HttpHost("example.com", 80));
DefaultRoutePlanner routePlanner = new DefaultRoutePlanner(DefaultSchemePortResolver.INSTANCE) {
#Override
public HttpRoute determineRoute(
final HttpHost host, final HttpRequest request, final HttpContext context) throws HttpException {
final HttpRoute route = super.determineRoute(host, request, context);
if (sitesToBlock.contains(route.getTargetHost())) {
throw new HttpException("Connection to " + host + " is blocked");
}
return route;
}
};
CloseableHttpClient client = HttpClients.custom()
.setRoutePlanner(routePlanner)
.build();
}
Is there an easier way to setup the http client for preemptive basic authentication than what described here?
In previous version (3.x) it used to be a simple method call (eg, httpClient.getParams().setAuthenticationPreemptive(true)).
The main thing I want to avoid is adding the BasicHttpContext to each method I execute.
If you are looking to force HttpClient 4 to authenticate with a single request, the following will work:
String username = ...
String password = ...
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(username, password);
HttpRequest request = ...
request.addHeader(new BasicScheme().authenticate(creds, request));
It's difficult to do this without passing a context through every time, but you can probably do it by using a request interceptor. Here is some code that we use (found from their JIRA, iirc):
// Pre-emptive authentication to speed things up
BasicHttpContext localContext = new BasicHttpContext();
BasicScheme basicAuth = new BasicScheme();
localContext.setAttribute("preemptive-auth", basicAuth);
httpClient.addRequestInterceptor(new PreemptiveAuthInterceptor(), 0);
(...)
static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(ClientContext.TARGET_AUTH_STATE);
// If no auth scheme avaialble yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
AuthScheme authScheme = (AuthScheme) context.getAttribute("preemptive-auth");
CredentialsProvider credsProvider = (CredentialsProvider) context.getAttribute(ClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(ExecutionContext.HTTP_TARGET_HOST);
if (authScheme != null) {
Credentials creds = credsProvider.getCredentials(new AuthScope(targetHost.getHostName(), targetHost.getPort()));
if (creds == null) {
throw new HttpException("No credentials for preemptive authentication");
}
authState.setAuthScheme(authScheme);
authState.setCredentials(creds);
}
}
}
}
This is the same solution as Mat's Mannion's, but you don't have to put localContext to each request. It's simpler, but it adds authentication to ALL requests. Useful, if you don't have control over individual requests, as in my case when using Apache Solr, which uses HttpClient internally.
import org.apache.http.HttpException;
import org.apache.http.HttpHost;
import org.apache.http.HttpRequest;
import org.apache.http.HttpRequestInterceptor;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.AuthState;
import org.apache.http.auth.Credentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.auth.BasicScheme;
import org.apache.http.protocol.HttpContext;
import org.apache.http.protocol.HttpCoreContext;
httpClient.addRequestInterceptor(new PreemptiveAuthInterceptor(), 0);
(...)
static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
CredentialsProvider credsProvider = (CredentialsProvider) context.getAttribute(HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
Credentials creds = credsProvider.getCredentials(new AuthScope(targetHost.getHostName(), targetHost.getPort()));
if (creds == null) {
throw new HttpException("No credentials for preemptive authentication");
}
authState.update(new BasicScheme(), creds);
}
}
}
Of course, you have to set the credentials provider:
httpClient.getCredentialsProvider().setCredentials(
new AuthScope(url.getHost(), url.getPort()),
new UsernamePasswordCredentials(username, password))
The AuthScope must not contain realm, as it is not known in advance.
A lot of the answers above use deprecated code. I am using Apache SOLRJ version 5.0.0.
My code consists of
private HttpSolrClient solrClient;
private void initialiseSOLRClient() {
URL solrURL = null;
try {
solrURL = new URL(urlString);
} catch (MalformedURLException e) {
LOG.error("Cannot parse the SOLR URL!!" + urlString);
throw new SystemException("Cannot parse the SOLR URL!! " + urlString, e);
}
String host = solrURL.getHost();
int port = solrURL.getPort();
AuthScope authScope = new AuthScope(host, port);
BasicTextEncryptor textEncryptor = new BasicTextEncryptor();
textEncryptor.setPassword("red bananas in the spring");
String decryptPass = textEncryptor.decrypt(pass);
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(userName, decryptPass);
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
authScope,
creds);
HttpClientBuilder builder = HttpClientBuilder.create();
builder.addInterceptorFirst(new PreemptiveAuthInterceptor());
builder.setDefaultCredentialsProvider(credsProvider);
CloseableHttpClient httpClient = builder.build();
solrClient = new HttpSolrClient(urlString, httpClient);
}
The PreemptiveAuthInterceptor is now as follows:-
static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
CredentialsProvider credsProvider = (CredentialsProvider)
context.getAttribute(HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
AuthScope authScope = new AuthScope(targetHost.getHostName(), targetHost.getPort());
Credentials creds = credsProvider.getCredentials(authScope);
if(creds == null){
}
authState.update(new BasicScheme(), creds);
}
}
}
A little late to the party but I came accross the thread trying to solve this for proxy pre-authorization of a post request. To add to Adam's response, I found the following worked for me:
HttpPost httppost = new HttpPost(url);
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(username, password);
Header bs = new BasicScheme().authenticate(creds, httppost);
httppost.addHeader("Proxy-Authorization", bs.getValue());
Thought that might be helpful for anyone else who runs into this.
I think the best way may be to just do it manually. I added the following function
Classic Java:
import javax.xml.bind.DatatypeConverter;
...
private static void addAuthHeader(HttpRequestBase http, String username, String password) throws UnsupportedEncodingException {
String encoded = DatatypeConverter.printBase64Binary((username + ":" + password).getBytes("UTF-8"));
http.addHeader("AUTHORIZATION", "Basic " + encoded);
}
HTTPRequestBase can be an instance of HttpGet or HttpPost
Android:
import android.util.Base64;
...
private static void addAuthHeader(HttpRequestBase http, String username, String password) throws UnsupportedEncodingException {
String encoded = Base64.encodeToString((username + ":" + password).getBytes("UTF-8"), Base64.NO_WRAP);
http.addHeader("AUTHORIZATION", "Basic " + encoded);
}
I'm using this code, based on my reading of the HTTPClient 4.5 docs:
HttpClientContext ctx = HttpClientContext.create()
ctx.setCredentialsProvider(new BasicCredentialsProvider())
ctx.setAuthCache(new BasicAuthCache())
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(user, pass)
AuthScope authScope = new AuthScope(host, port)
ctx.getCredentialsProvider.setCredentials(authScope, credentials)
// This part makes authentication preemptive:
HttpHost targetHost = new HttpHost(host, port, scheme)
ctx.getAuthCache.put(targetHost, new BasicScheme())
...and make sure you always pass that context to HTTPClient.execute().
I don't quite get your closing comment. It's the HttpClient that has all of that machinery for doing preemptive auth, and you only have to do that once (when you construct and configure your HttpClient). Once you've done that, you construct your method instances the same way as always. You don't "add the BasicHttpContext" to the method.
Your best bet, I'd think, is to have your own object that sets up all of the junk required for preemptive auth, and has a simple method or methods for executing requests on given HTTPMethod objects.
in android,Mat Mannion's answer can't resolve https,still send two requests,you can do like below,the trick is append authHeader with user-agent：
public static DefaultHttpClient createProxyHttpClient() {
try {
final DefaultHttpClient client = createPlaintHttpClient();
client.setRoutePlanner(new HttpRoutePlanner() {
#Override
public HttpRoute determineRoute(HttpHost target, HttpRequest request, HttpContext context) throws HttpException {
boolean isSecure = "https".equalsIgnoreCase(target.getSchemeName());
if (needProxy) {
Header header = isSecure ? ProxyUtils.createHttpsAuthHeader() : ProxyUtils.createAuthHeader();
if (isSecure) {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT + "\r\n" + header.getName() + ":" + header.getValue());
} else {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT);
if (request instanceof RequestWrapper) {
request = ((RequestWrapper) request).getOriginal();
}
request.setHeader(header);
}
String host = isSecure ? ProxyUtils.SECURE_HOST : ProxyUtils.HOST;
int port = isSecure ? ProxyUtils.SECURE_PORT : ProxyUtils.PORT;
return new HttpRoute(target, null, new HttpHost(host, port), isSecure);
} else {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT);
return new HttpRoute(target, null, isSecure);
}
}
});
return client;
} catch (Exception e) {
e.printStackTrace();
return new DefaultHttpClient();
}
}
public static DefaultHttpClient createPlaintHttpClient() {
try {
KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
trustStore.load(null, null);
PlainSSLSocketFactory socketFactory = new PlainSSLSocketFactory(trustStore);
socketFactory.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
BasicHttpParams params = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(params, 30000);
HttpConnectionParams.setSoTimeout(params, 30000);
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
HttpProtocolParams.setContentCharset(params, HTTP.UTF_8);
SchemeRegistry registry = new SchemeRegistry();
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));
registry.register(new Scheme("https", socketFactory, 443));
ThreadSafeClientConnManager ccm = new ThreadSafeClientConnManager(params, registry);
HttpClientParams.setCookiePolicy(params, CookiePolicy.BROWSER_COMPATIBILITY);
final DefaultHttpClient client = new DefaultHttpClient(ccm, params);
client.setRoutePlanner(new HttpRoutePlanner() {
#Override
public HttpRoute determineRoute(HttpHost target, HttpRequest arg1, HttpContext arg2) throws HttpException {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT);
return new HttpRoute(target, null, "https".equalsIgnoreCase(target.getSchemeName()));
}
});
return client;
} catch (Exception e) {
e.printStackTrace();
return new DefaultHttpClient();
}
}
SolrConfig:
#Configuration
public class SolrConfig {
#Value("${solr.http.url}")
private String solrUrl;
#Value("${solr.http.username}")
private String solrUser;
#Value("${solr.http.password}")
private String solrPassword;
#Value("${solr.http.pool.maxTotal}")
private int poolMaxTotal;
#Value("${solr.http.pool.maxPerRoute}")
private int pollMaxPerRoute;
#Bean
public SolrClient solrClient() {
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(poolMaxTotal);
connectionManager.setDefaultMaxPerRoute(pollMaxPerRoute);
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(solrUser, solrPassword));
CloseableHttpClient httpClient = HttpClientBuilder.create()
.addInterceptorFirst(new PreemptiveAuthInterceptor())
.setConnectionManager(connectionManager)
.setDefaultCredentialsProvider(credentialsProvider)
.build();
return new HttpSolrClient.Builder(solrUrl).withHttpClient(httpClient).build();
}
}
PreemptiveAuthInterceptor:
public class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context)
throws HttpException {
AuthState authState = (AuthState) context
.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
CredentialsProvider credentialsProvider = (CredentialsProvider) context
.getAttribute(HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context
.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
Credentials credentials = credentialsProvider.getCredentials(new AuthScope(
targetHost.getHostName(), targetHost.getPort()));
if (credentials == null) {
throw new HttpException(
"No credentials for preemptive authentication");
}
authState.update(new BasicScheme(), credentials);
}
}
}
I would like to migrate from Commons HttpClient (3.x) to HttpComponents Client (4.x) but having difficulty how to handle redirects. The code works properly under Commons HttpClient but breaks when migrated to HttpComponents Client. Some of the links get undesirable redirects but when I set "http.protocol.handle-redirects" to 'false' a large number links stop working altogether.
Commons HttpClient 3.x:
private static HttpClient httpClient = null;
private static MultiThreadedHttpConnectionManager connectionManager = null;
private static final long MAX_CONNECTION_IDLE_TIME = 60000; // milliseconds
static {
//HttpURLConnection.setFollowRedirects(true);
CookieManager manager = new CookieManager();
manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(manager);
connectionManager = new MultiThreadedHttpConnectionManager();
connectionManager.getParams().setDefaultMaxConnectionsPerHost(1000); // will need to set from properties file
connectionManager.getParams().setMaxTotalConnections(1000);
httpClient = new HttpClient(connectionManager);
}
/*
* Retrieve HTML
*/
public String fetchURL(String url) throws IOException{
if ( StringUtils.isEmpty(url) )
return null;
GetMethod getMethod = new GetMethod(url);
HttpClient httpClient = new HttpClient();
//configureMethod(getMethod);
//ObjectInputStream oin = null;
InputStream in = null;
int code = -1;
String html = "";
String lastModified = null;
try {
code = httpClient.executeMethod(getMethod);
in = getMethod.getResponseBodyAsStream();
//oin = new ObjectInputStream(in);
//html = getMethod.getResponseBodyAsString();
html = CharStreams.toString(new InputStreamReader(in));
}
catch (Exception except) {
}
finally {
try {
//oin.close();
in.close();
}
catch (Exception except) {}
getMethod.releaseConnection();
connectionManager.closeIdleConnections(MAX_CONNECTION_IDLE_TIME);
}
if (code <= 400){
return html.replaceAll("\\s+", " ");
} else {
throw new Exception("URL: " + url + " returned response code " + code);
}
}
HttpComponents Client 4.x :
private static HttpClient httpClient = null;
private static HttpParams params = null;
//private static MultiThreadedHttpConnectionManager connectionManager = null;
private static ThreadSafeClientConnManager connectionManager = null;
private static final int MAX_CONNECTION_IDLE_TIME = 60000; // milliseconds
static {
//HttpURLConnection.setFollowRedirects(true);
CookieManager manager = new CookieManager();
manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(manager);
connectionManager = new ThreadSafeClientConnManager();
connectionManager.setDefaultMaxPerRoute(1000); // will need to set from properties file
connectionManager.setMaxTotal(1000);
httpClient = new DefaultHttpClient(connectionManager);
// HTTP parameters stores header etc.
params = new BasicHttpParams();
params.setParameter("http.protocol.handle-redirects",false);
}
/*
* Retrieve HTML
*/
public String fetchURL(String url) throws IOException{
if ( StringUtils.isEmpty(url) )
return null;
InputStream in = null;
//int code = -1;
String html = "";
// Prepare a request object
HttpGet httpget = new HttpGet(url);
httpget.setParams(params);
// Execute the request
HttpResponse response = httpClient.execute(httpget);
// The response status
//System.out.println(response.getStatusLine());
int code = response.getStatusLine().getStatusCode();
// Get hold of the response entity
HttpEntity entity = response.getEntity();
// If the response does not enclose an entity, there is no need
// to worry about connection release
if (entity != null) {
try {
//code = httpClient.executeMethod(getMethod);
//in = getMethod.getResponseBodyAsStream();
in = entity.getContent();
html = CharStreams.toString(new InputStreamReader(in));
}
catch (Exception except) {
throw new Exception("URL: " + url + " returned response code " + code);
}
finally {
try {
//oin.close();
in.close();
}
catch (Exception except) {}
//getMethod.releaseConnection();
connectionManager.closeIdleConnections(MAX_CONNECTION_IDLE_TIME, TimeUnit.MILLISECONDS);
connectionManager.closeExpiredConnections();
}
}
if (code <= 400){
return html;
} else {
throw new Exception("URL: " + url + " returned response code " + code);
}
}
I won't want redirects but under HttpClient 4.x if I enable redirects then I get some that are undesirable, e.g. http://www.walmart.com/ => http://mobile.walmart.com/. Under HttpClient 3.x no such redirects happens.
What do I need to do to migrate HttpClient 3.x to HttpClient 4.x without breaking the code?
It is not the issue with HttpClient 4.x, might be the way target server handle the request, since the user agent is httpclient, it may be handled as mobile (target server may consider other than available browsers like, i.e, chrome, mozilla etc as mobile.)
Please use below code to set it manually
httpclient.getParams().setParameter(
org.apache.http.params.HttpProtocolParams.USER_AGENT,
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2"
);
I use Apache HttpClient to first Request a page for the Cookies, and then post to a page with those Cookies. To be able to get the second page, the Cookie must be sent with the post. I've read somewhere that HttpClient automatically saves and Sends the needed Cookies, but somehow I keep stuck at the first page, probably due to Cookies not being get properly, or not being sent properly.
public class Main {
static BufferedWriter writer;
public static void main(String args[]) throws ClientProtocolException, IOException {
getRequest();
}
public static void getRequest() throws ClientProtocolException, IOException {
HttpClient client = new DefaultHttpClient();
//the request to get the Cookies
HttpGet request = new HttpGet("http://www.SiteNameCutOut.cz");
List <NameValuePair> parameters = new ArrayList <NameValuePair>();
parameters.add(new BasicNameValuePair("view_state", "eaftOTAPef3NDs79"));
parameters.add(new BasicNameValuePair("age", "23"));
parameters.add(new BasicNameValuePair("button", "go"));
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(parameters);
HttpPost post = new HttpPost("http://www.SameSiteAsAbove.cz");
post.setEntity(entity);
//post.addHeader(request.getFirstHeader("Set-Cookie")); maybe?
post.addHeader("Host","theSiteHost");
post.addHeader("User-Agent","Mozilla/5.0 (Windows NT 6.0; rv:2.0.1) Gecko/20100101 Firefox/4.0.1");
post.addHeader("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
post.addHeader("Accept-Language","en-us,en;q=0.5");
post.addHeader("Accept-Encoding","gzip, deflate");
post.addHeader("Accept-Charset","ISO-8859-1,utf-8;q=0.7,*;q=0.7");
post.addHeader("Keep-Alive","115");
post.addHeader("Connection","keep-alive");
client.execute(request);
try {
request.abort();
HttpResponse response = client.execute(post);
writer = new BufferedWriter(new FileWriter("test001.html"));
writer.write(HttpHelper.request(response)); //gets html of the response
} catch (IOException ex) {
System.out.println("**Error**");
} finally {
if(writer != null){
writer.close();
}
else{
System.out.println("Writer is null");
}
}
}
}
So i hope anyone can help me, Thanks !
You should indicate how to manage the cookies like this:
request.getParams().setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.BROWSER_COMPATIBILITY);