How make logs go into log stash log view - java

this is my conf. file
input {
file {
type => "java"
path => "/UUUU********/IdeaProjects/elk-stack-logging-example/elk-example.log"
codec => multiline {
pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
negate => "true"
what => "previous"
}
}
}
filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
}
}
grok {
match => [ "message",
"(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- \[(?<thread>[A-Za-z0-9-]+)\] [A-Za-z0-9.]*\.(?<class>[A-Za-z0-9#_]+)\s*:\s+(?<logmessage>.*)",
"message",
"(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(?<logmessage>.*)"
]
}
date {
match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
}
}
output {
# Sending properly parsed log events to elasticsearch
elasticsearch {
hosts => ['https://**********************:9243/']
user => 'elastic'
password => '*********************'
index => "logstash_%{+YYYYMMdd}"
}
stdout { codec => rubydebug }
}
after running the code I go to elastic search then I want to create a data view inserting the name log stash it says not data stream or index found for a given name.
How to make this conf file work and create that index so I can create the view

Related

Karate DSL: Convert String to Json and insert value? [duplicate]

I Have a file json named production_2.json
[
{
"v":{
"id":"rep_01564526",
"name":"tuttoverde.pgmx",
"type":"PRODUCTION_STARTED",
"ute":"CDL",
"ver":"1.0",
"status":"EXE"
},
"ts":"2020-11-19T08:00:00Z"
},
{
"v":{
"id":"rep_01564526",
"name":"tuttoverde.pgmx",
"type":"PRODUCTION_ENDED",
"ute":"CDL",
"ver":"1.0",
"status":"EXE"
},
"ts":"2020-11-19T17:00:00Z"
}
]
And have the folling Karate code, that:
Read the file production_2.json
and for each element of the array send a topic
I * def sendtopics =
"""
function(i){
var topic = "data." + machineID + ".Production";
var payload = productions[i];
karate.log('topic: ', topic )
karate.log('payload: ', payload )
return mqtt.sendMessage(payload, topic);
}
"""
* def productions = read('this:productions_json/production_2.json')
* def totalProductionEvents = productions.length
* def isTopicWasSent = karate.repeat(totalProductionEvents, sendtopics)
* match each isTopicWasSent == true
The function
mqtt.sendMessage(payload, topic);
is a function in java, that have the following segnature
public Boolean sendMessage(String payload, String topic) {
System.out.println("Publishing message: ");
System.out.println("payload " + payload);
System.out.println("topic " + topic);
return true;
}
the problem is that the value of the "payload" inside the javascript function is correct and is printed correctly, while inside the "sendMessage" function the value of the payload is formatted incorrectly.
For example here is what it prints inside karate.log('payload: ', payload )
payload: {
"v": {
"id": "rep_01564526",
"name": "tuttoverde.pgmx",
"type": "PRODUCTION_STARTED",
"ute": "CDL",
"ver": "1.0",
"status": "EXE"
},
"ts": "2021-01-08T08:00:00Z"
}
And Here instead what is printed on the function "sendMessage" of the java class
payload {v={id=rep_01564526, name=tuttoverde.pgmx, type=PRODUCTION_STARTED, ute=CDL, ver=1.0, status=EXE, ts=2021-01-08T08:00:00Z}
I don't understand why the payload is formatted incorrectly (= instead of : ) and is it not a string. I also tried using the following solution and it doesn't work for me
* def sendtopics =
"""
function(i){
var topic = "data." + machineID + ".Production";
var payload = productions[i];
var payload2 = JSON.stringify(payload);
return mqtt.sendMessage(payload2, topic);
}
"""
How do I convert an object inside javascript to a string so I can pass it to java?
You are doing some really advanced stuff in Karate. I strongly suggest you start looking at the new version (close to release) and you can find details here: https://github.com/intuit/karate/wiki/1.0-upgrade-guide
The reason is because the async and Java interop has some breaking changes - and there are some new API-s you can call on the karate object in JS to do format conversions:
var temp = karate.fromString(payload);
And karate.log() should work better and not give you that odd formatting you are complaining about. With the current version you can try karate.toJson() if that gives you the conversion you expect.
Given your advanced usage, I recommend you start using the new version and provide feedback on anything that may be still needed.

Create PHP array with the same key name

i have created an api in java that accepts (name="sample" multipart[]) and when i'm requesting from php curl,
curl file objects in the array overwrites the old one to new, it means, only one value in the array is accepted.
This is one of the code that i tried:
$files = array(
'path/sample1.png',
'path/sample2.png',
);
$postfields = array();
foreach ($files as $index => $file) {
if (function_exists('curl_file_create')) {
$file = new CURLFILE(realpath($file), mime_content_type($file), basename($file));
}
else {
$file = '#' . realpath($file);
}
$postfields["sample"] = $file;
}
Note: I already tried sending a request in postman and it works perfectly with same keyname. The generated code have the same keyname in the post fields like this..
"sample" => "",
"sample" => ""
I don't know what to follow since the postman request is success, i tried also using the generated code from postman but the output is the same as stated above.
CURL:
CURLOPT_HEADER => 0,
CURLOPT_VERBOSE => 0,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SAFE_UPLOAD => true,
CURLOPT_POST => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => 0,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",

React Js : Passing a condition to a state value based on another state value

I have been scattering the internet in search of an answer but my guess is that what i am doing is either wrong or i don't know the right therm because i can't find any relevant answer. Long story short, i want to pass a condition to a state value that is based on another state value inside the constructor. Why? Because i am building a "Form Generator" and i want to be able to link an input to another one. Exemple : If a checked box is checked, then you can enter text inside a text input otherwise it is disabled. Pretty simple right? If anybody can just point me in a direction that would be great. So far this is what i have tried but it doesn't work.
constructor(props){
this.state =
Inputs : [
//Theres a bunch of inputs here
{
name : "Icon",
group: 1,
width: 6,
type: "select",
label: "Icon du menu",
disabled: () =>(
!this.state.Inputs[0].value //I want to link the disable property to an
//input value
),
value: ""
}
]
}
I have also tried to create a constraint object in my state and iterate trough the value during the mounting process but i found it to be too fragile and time consuming. If i could directly create the link inside the input object that would be great.
Thanks for your time and great wisdom!
-Sam
You'll want the initial index [0] to have a different disabled function than !this.state.Inputs[0].value or else the disabled property of all inputs will be stuck and never change.
Here is a demonstration on how to link the disabled property to the input value of another:
class App extends React.Component {
constructor(props){
super(props);
this.state = {
inputs: [
{
name : "Form1",
disabled: () => { return false; },
value: ""
},
{
name : "Form2",
disabled: () => { return this.state.inputs[0].value === ""; },
value: ""
},
{
name : "Form3",
disabled: () => { return this.state.inputs[0].value === ""; },
value: ""
}
]
};
}
onInputChange = (e, index) => {
const newInputs = this.state.inputs;
newInputs[index].value = e.target.value;
this.setState({ inputs: newInputs });
};
render() {
return (
<div>
{this.state.inputs &&
this.state.inputs.map((inputItem, index) => {
return (
<div>
{this.state.inputs[index].name}:
<input onChange={(e) => { this.onInputChange(e, index); }} disabled={this.state.inputs[index].disabled()}/>
</div>
);
})}
</div>
);
}
}
// Render it
ReactDOM.render(
<App/>,
document.getElementById("react")
);
<div id="react"></div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react/15.1.0/react.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/react/15.1.0/react-dom.min.js"></script>
In the demonstration if you have a value, other than an empty string, for the first input, it will unlock the rest of the inputs for user input.

Logstash: Is there a way to invoke grok templates on the basis of field name

I am using ELK with logstash-logback-encoder for pushing logs to the Logstash. Now I want to use the same stack i.e ELK with logstash-logback-encoder for analytics.
Flow:
API(Create User)----> Commit data to RDBMS ----->
Callback Listener(on post persist and post update) --->
Logger.info("IndexName: {} . DocId: {} .User json: {}", "Customer", user.getID(), user.getJson());
Logger.info(); logstash-logback-encoder, will push the data to Logstash, which will push the data to ES.
My logstash.conf is as below:
input {
tcp {
port => 5044
codec => multiline {
what => "previous"
}
}
}
filter{
grok {
match => ["message", "(?<index_name>(?<=IndexName: ).*?(?=\s))"]
match => ["message", "(?<doc_id>(?<=DocId: ).*?(?=\s))"]
break_on_match => false
remove_tag => ["_grokparsefailure","multiline"]
}
mutate {
gsub => ['message', "\t", " "]
gsub => ['message',"\e\[(\d*;)*(\d*)m"," "]
}
}
output {
if [index_name] == "Customer" {
elasticsearch {
hosts => ["localhost:9200"]
index => "analytics-customers"
document_id => "%{doc_id}"
}
}else {
elasticsearch {
hosts => ["localhost:9200"]
}
}
stdout { codec => rubydebug }
}
My problem is that if I want to use Logstash for analytics then I have to parse the json using grok. With the amount of table and fields that I have, the logstash.conf will became really huge.
Is there a way by which I can apply grok templates in logstash.conf which I can invoke on the basis of index name.
Like:
grok {
match => ["message", "(?<index_name>(?<=IndexName: ).*?(?=\s))"]
if(index_name=="User"){
//Invoke User template which will fetch/create fields from passed json.
}
if(index_name=="Order"){
//Invoke Order template which will fetch/create fields from passed json.
}
}
If you can manage to get the logs in a single line, it would be the nicest way. Because than you can change the codec to "json_lines" and everything is parsed automatically!
Otherwise you can use IF (described here )
Example:
if [subsystem] == "http" {
mutate{ ... }
grok{ ... }
}

Puppet - Remove ALL versions of Java from PATH

I am attempting to remove ALL version of java from windows PATH via Puppet.
I will replace it with %JAVA_HOME%\bin.
Here is the current code:
include windows_path
define obsolete_java {
windows_path {
$name:
ensure => absent,
directory => "C:\\Program Files\\Java\\$name\\bin";
}
}
define windows_java::setup (
$ensure = 'present',
$source = undef,
$file = undef,
$package = undef ) {
case $::osfamily {
Windows : { $supported = true }
default : { fail("The ${module_name} module is not supported on ${::osfamily} based systems") }
}
# Validate parameters
if ($source == undef) {
fail('source parameter must be set')
}
if ($file == undef) {
fail('file parameter must be set')
}
if ($package == undef) {
fail('package parameter must be set')
}
# Validate input values for $ensure
if !($ensure in ['present', 'absent']) {
fail('ensure must either be present or absent')
}
if ($caller_module_name == undef) {
$mod_name = $module_name
} else {
$mod_name = $caller_module_name
}
if ($ensure == 'present'){
# ensures main directory exists
file {'C:\Program Files\Java':
ensure => directory,
}
# copies source executable over
file { "C:\\Program Files\\Java\\$source":
ensure => present,
source => "puppet:///extra_files/java/windows/$source",
before => Package["$package"],
source_permissions => ignore,
}
# Name of package must match name when installed
package { "$package":
ensure => installed,
source => "C:\\Program Files\\java\\$source",
install_options => '/s',
}
# sets JAVA_HOME. If already existant, replaces it.
windows_env { "JAVA_HOME=C:\\Program Files\\Java\\$file":
mergemode => clobber,
}
# removes previous Java from path
obsolete_java {'jdk1.7.0_51':}
# Adds JAVA_HOME to the path.
windows_path {'javaPath':
ensure => present,
directory => "%JAVA_PATH%\bin",
}
} else {
package { 'remove-package':
name => "$package",
ensure => absent,
}
windows_env { 'JAVA_HOME':
ensure => 'absent',
mergemode => clobber,
}
windows_path {'remove_java_path':
ensure => absent,
directory => "C:\\Program Files\\Java\\$file\\bin",
}
}
}
Specifically:
define obsolete_java {
windows_path {
$name:
ensure => absent,
directory => "C:\\Program Files\\Java\\$name\\bin";
}
}
called here:
# removes previous Java from path
obsolete_java {'jdk1.7.0_51':}
Is it possible just have it remove anything it finds under: C:\Program Files\Java\any_jdk
bin ?
Edit: More importantly, can I force it to find any directory that contains the word "java" and remove it from PATH?
Seeing as the windows_path type is somewhat complex, there seems to be no conceivable way for the provider to prefetch the full list of existing paths. You will therefor be unable to purge the java paths.
That leaves you with only one option that I can see: Create an external fact (or plain custom fact) to retrieve the list of directories in the PATH. If you don't use Facter 2, you will need to get the delimited PATH string and process it on the master. Facter 2 can hand an array to the master.
If you have them in a variable, say $path_list, you can hand them to a similar defined type as the one you are using already.
filter_path { $path_list: }
define filter_path {
if $name =~ /java/ {
windows_path { $name: ensure => absent, directory => $name }
}
}

Categories

Resources