Develop Reference

log4j gets stuck at a maximum number of 13 files even though the configured limit is 50?

This is the log4j properties file we are using, could you please explain why the limit for number of files is stuck at 13 instead of 50 as defined in "maxIndex" Property?
# Root logger option
log4j.rootLogger=INFO,file,stdout
log4j.logger.com.cisco.assurance=DEBUG, file
# Redirect log messages to console
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.Threshold=DEBUG
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd'T'HH:mm:ss.SZ} %-5p %m%n
# Redirect log messages to a log file
log4j.appender.file=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.file.rollingPolicy.FileNamePattern=/var/log/smeu/AssuranceMS-%i.log.gz
log4j.appender.file.rollingPolicy.ActiveFileName=/var/log/smeu/AssuranceMS.log
log4j.appender.file.triggeringPolicy.MaxFileSize=204800000
log4j.appender.file.rollingPolicy.maxIndex=50
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d [%t] %-5p (%F:%L) - %m%n
log4j.appender.file.Threshold=DEBUG

From the API of org.apache.log4j.rolling.FixedWindowRollingPolicy, it is reduced by design to 12 backups plus the current one, so 13 maximum.
You can look at it : https://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/FixedWindowRollingPolicy.html

This is happening because large window size is discouraged and the maxIndex is automatically reduced to 12. It is well documented here.
Given that this rollover algorithm requires as many file renaming operations as the window size, large window sizes are discouraged. The current implementation will automatically reduce the window size to 12 when larger values are specified by the user.

How to write Hebrew strings to a log file using log4j

How to write a Hebrew string to a log4j file. Right now I see ?????? in the file.
I have searched everywhere online to convert Unicode to string:
String abc = myStr.replaceAll("\u200F", "");
abc = abc.replaceAll("\u200E", "");
byte[] utf8Bytes = abc.getBytes(Charset.forName("UTF-8"));
String value = new String(utf8Bytes);
log.debug("value : "+ value );
I just need to write out a Hebrew string to a Log4j file in a readable format. Here is my configuration:
log4j.rootLogger=debug, stdout, R log4j.logger.testlogging=DEBUG
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd} %5p [%t] (%F:%L) - %m%n log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=C:\\dri\\ums.log log4j.appender.R.MaxBackupIndex=5
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern= %d{dd MMM yyyy HH:mm:ss,SSS} %5p [%t] (%F:%L) - %m%n log4j.appender.FILE.encoding=UTF-8

Based from what I've gathered from the comments and my own experience this is most probably not an issue with Log4j itself. I've posted a comment indicating just that:
What exactly do you mean with Log4j file? Is it a regular text log file that the FileAppender points to? Because I've tried printing Hebrew text right now and all is working out fine. I believe this is not a Log4j issue and might be related to your text reader.
Other comments have confirmed their suspicion that it is your text reader that might be causing this issue. I was able to reproduce your issue by doing the following in Notepad++:
Open a new tab in notepad++.
Copy and paste sample text containing Hebrew letters.
Language -> Convert to ANSI
Text before the conversion:
See also: אלף־בית‎ and אַלף־בית‎
Text after conversion:
See also: ???????? and ?????????
Based on the code you provided (assuming there is no shenanigans that we don't know for behind the scenes) we can definitively conclude you are either writing to a file that has it's encoding set to ANSI where all your special characters are being converted to question marks because they cannot be decoded or your characters are being read as UTF-8 but merely displayed as ANSI.
ANSI and UTF-8 are both encoding formats. ANSI is the common one byte format used to encode Latin alphabet; whereas, UTF-8 is a Unicode format of variable length (from 1 to 4 bytes) which can encode all possible characters.
I would recommend following these steps:
Navigate to Settings -> Preferences -> New Document -> Encoding and make sure that the UTF-8 (Apply to opened ANSI files) option is selected.
Close all your files currently opened in Notepad++ and delete the log file. Make sure you are actually closing the files instead of just closing Notepad++. This should clear the file entries from cache and allow you to open them again with a different encoding.
Run your Java application and let Log4j print to the file.
Open the file with Notepad++ and check that you are encoding in UTF-8 by clicking on Encoding tab. If the option is not set to UTF-8, change it.
If none of the above worked, please post further information in the comments.
Unfortunately I am not that well versed in encoding matters and had to look some stuff up in the process of writing this, so I can't help you as much as I would like to. However in addition to providing the steps above I can direct you to the following links which should provide you with further knowledge and (consequently) more insight into your problem:
Encodings And Character Display - Notepad++ Wiki
ANSI to UTF-8 in Notepad++ - SuperUser

Encode pattern layout in apache log4j

Trying to use the following pattern layout to print encoded log messages, so that \n will not be printed as a new line in the logs.
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{MM-dd-yyyy HH:mm:ss,SSS} %5p [%t] (%F:%L) - %enc{%m}%n"/>
</layout>
But it literally prints %enc{ before every message and prints } at the end of the messages, instead of encoding them. \n is also not being replaced.
Example: For this code logger.info("my \n message") following is the output log
08-01-2018 01:26:37,642 INFO [localhost-startStop-1] (ClassName1.java:53) - %enc{my
message}
instead it should be like,
08-01-2018 01:26:37,642 INFO [localhost-startStop-1] (ClassName1.java:53) - my message
Followed this document https://logging.apache.org/log4j/log4j-2.1/manual/layouts.html But I'm not sure what is wrong with the pattern.

You should determine the encoding format in curly brackets. For example %enc{%m}{HTML} encodes the message in HTML format. You could also use CRLF: enc{%m}{CRLF}

Log4j to write json array to disk

Im creating a log file system. the log file will be in json format so a server can read it after but i dont think thats too important. What i need to know is can log4j be configured to write into to a file but without any tags like info,debug, timestamp etc in the file. I have looked here
but this polutes the file with with other things. I want ONLY the data i write to show up in the file. I'd also like to set some kind of file rotation on the file if it gets too big after a max size is reached.

This is relatively easy, using a log4j.properties configuration file (place it at the top of your classpath, and Log4j will 'just find it'):
# This is the default logger, simply logs to console
log4j.logger.com.foo.bar=DEBUG,A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
# Note the Pattern here, emits a lot of stuff - btw, don't use this in production
# %C is expensive - see the Javadoc for ConversionPattern for the meaning of all
# the % modifiers:
log4j.appender.A1.layout.ConversionPattern=%d{MMM dd, HH:mm:ss} [%C{2}] %-5p - %m%n
# Logging to file can be enabled by using this one
log4j.logger.com.example=DEBUG, R
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=/var/log/generic.log
log4j.appender.R.MaxFileSize=100KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=1
# This is the most minimalist layout you can have: just the 'm'essage is emitted
# (and a \n newline):
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%m%n
All the classes in the com.foo.bar package (and subpackages) will log to console, those in com.example (and below) will log to /var/log/generic.log.
To emit JSON, just use Jackson (com.fasterxml) convert your data to a JSON object and write it out as a string.

What you want is the PatternLayout with %m%n only, and combine with the answer to previously asked question here

You should be able to write a custom log appender, I have a vague recollection of there being an example to do this with MongoDB.
MongoDD stores its data as JSON, so it have converted the log to json format before inserting it.
I would start by searching for a MondgoDB appender, and looking at the file appender that is shipped, that should give you a starting point for an appender if one doesnt already exist.
http://log4mongo.org/display/PUB/Home

Find and replace words/lines in a file

I have a file (more specifically, a log4j configuration file) and I want to be able to read in the file and pick out certain lines in the code and replace them. For example, within the file there is a string of text that indicates the directory it is stored in, or the level of the logger. I want to be able to replace those string of text without reading in the file, writing it to another file, and deleting the original file. Is there a more efficient way of doing find and replace texts in a file using Java?
Here is an example of the text file I'm trying to work with:
log4j.rootLogger=DEBUG, A0
log4j.appender.A0=org.apache.log4j.RollingFileAppender
log4j.appender.A0.File=C:/log.txt
log4j.appender.A0.MaxFileSize=100KB
log4j.appender.A0.MaxBackupIndex=1
log4j.appender.A0.layout=org.apache.log4j.RollingFileAppender
log4j.appender.A0.layout.ConversionPattern=%-4r [%t] %-5p: %c %x - %m%n
I want to be able to read the file and replace 'DEBUG' with another level or replace the file directory name 'C:/log.txt'. The log configuration file is also written in xml. An example of that is featured below.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration>
<appender class="org.apache.log4j.RollingFileAppender" name="A0">
<param name="append" value="false"/>
<param name="File" value="C:/log/.txt"/>
<param name="MaxBackupIndex" value="1"/>
<param name="MaxFileSize" value="100KB"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-4r [%t] %-5p: %c %x - %m%n"/>
</layout>
</appender>
<root>
<level value="DEBUG"/>
<appender-ref ref="A0"/>
</root>
</log4j:configuration>
I'm thinking it may be possible to use a hash map for this type of implementation?

Any decent text editor has a search&replace facility that supports regular expressions.
If however, you have reason to reinvent the wheel in Java, you can do:
Path path = Paths.get("test.txt");
Charset charset = StandardCharsets.UTF_8;
String content = new String(Files.readAllBytes(path), charset);
content = content.replaceAll("foo", "bar");
Files.write(path, content.getBytes(charset));
This only works for Java 7 or newer. If you are stuck on an older Java, you can do:
String content = IOUtils.toString(new FileInputStream(myfile), myencoding);
content = content.replaceAll(myPattern, myReplacement);
IOUtils.write(content, new FileOutputStream(myfile), myencoding);
In this case, you'll need to add error handling and close the streams after you are done with them.
IOUtils is documented at http://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/IOUtils.html

After visiting this question and noting the initial concerns of the chosen solution, I figured I'd contribute this one for those not using Java 7 which uses FileUtils instead of IOUtils from Apache Commons. The advantage here is that the readFileToString and the writeStringToFile handle the issue of closing the files for you automatically. (writeStringToFile doesn't document it but you can read the source). Hopefully this recipe simplifies things for anyone new coming to this problem.
try {
String content = FileUtils.readFileToString(new File("InputFile"), "UTF-8");
content = content.replaceAll("toReplace", "replacementString");
File tempFile = new File("OutputFile");
FileUtils.writeStringToFile(tempFile, content, "UTF-8");
} catch (IOException e) {
//Simple exception handling, replace with what's necessary for your use case!
throw new RuntimeException("Generating file failed", e);
}

public static void replaceFileString(String old, String new) throws IOException {
String fileName = Settings.getValue("fileDirectory");
FileInputStream fis = new FileInputStream(fileName);
String content = IOUtils.toString(fis, Charset.defaultCharset());
content = content.replaceAll(old, new);
FileOutputStream fos = new FileOutputStream(fileName);
IOUtils.write(content, new FileOutputStream(fileName), Charset.defaultCharset());
fis.close();
fos.close();
}
above is my implementation of Meriton's example that works for me. The fileName is the directory (ie. D:\utilities\settings.txt). I'm not sure what character set should be used, but I ran this code on a Windows XP machine just now and it did the trick without doing that temporary file creation and renaming stuff.

You might want to use Scanner to parse through and find the specific sections you want to modify. There's also Split and StringTokenizer that may work, but at the level you're working at Scanner might be what's needed.
Here's some additional info on what the difference is between them:
Scanner vs. StringTokenizer vs. String.Split

This is the sort of thing I'd normally use a scripting language for. It's very useful to have the ability to perform these sorts of transformations very simply using something like Ruby/Perl/Python (insert your favorite scripting language here).
I wouldn't normally use Java for this since it's too heavyweight in terms of development cycle/typing etc.
Note that if you want to be particular in manipulating XML, it's advisable to read the file as XML and manipulate it as such (the above scripting languages have very useful and simple APIs for doing this sort of work). A simple text search/replace can invalidate your file in terms of character encoding etc. As always, it depends on the complexity of your search/replace requirements.

You can use Java's Scanner class to parse words of a file and process them in your application, and then use a BufferedWriter or FileWriter to write back to the file, applying the changes.
I think there is a more efficient way of getting the iterator's position of the scanner at some point, in order to better implement editting. But since files are either open for reading, or writing, I'm not sure regarding that.
In any case, you can use libraries already available for parsing of XML files, which have all of this implemented already and will allow you to do what you want easily.