I'm using MyBatis in a project that fetches many rows (more than 2M rows).
I have a simple question about how MyBatis works. Everytime I need an action from the mapper, does MyBatis read the XML file and extract the query? Or are the mappers put into memory and MyBatis access them directly?
This is important, because access and read a XML file can have impact on the performance values we are expecting.
Thanks in advance.
Regards
Shortly, MyBatis parses the XML file when you first build your SqlSessionFactory from the configuration XML file. All the properties, mappers and settings are stored in-memory after that.
Explanation:
As stated in the doc, you can set up your SqlSessionFactory without XML, directly in Java as follows (see, last line):
DataSource dataSource = BlogDataSourceFactory.getBlogDataSource();
TransactionFactory transactionFactory = new JdbcTransactionFactory();
Environment environment = new Environment("development", transactionFactory, dataSource);
Configuration configuration = new Configuration(environment);
configuration.addMapper(BlogMapper.class);
SqlSessionFactory sqlSessionFactory = new SqlSessionFactoryBuilder().build(configuration);
Actually, when you build your SqlSessionFactory from XML, you will write something like this:
String resource = "org/mybatis/example/mybatis-config.xml";
InputStream inputStream = Resources.getResourceAsStream(resource);
SqlSessionFactory sqlSessionFactory = new SqlSessionFactoryBuilder().build(inputStream);
If you trace the source in SqlSessionFactoryBuilder,
public SqlSessionFactory build(Reader reader, String environment, Properties properties) {
try {
XMLConfigBuilder parser = new XMLConfigBuilder(reader, environment, properties);
return build(parser.parse());
...
The parse() method returns Configuration object, which holds all the information you supplied in the XML file.
public class Configuration {
protected Environment environment;
protected boolean safeRowBoundsEnabled = false;
protected boolean safeResultHandlerEnabled = true;
protected boolean mapUnderscoreToCamelCase = false;
protected boolean aggressiveLazyLoading = true;
protected boolean multipleResultSetsEnabled = true;
protected boolean useGeneratedKeys = false;
protected boolean useColumnLabel = true;
protected boolean cacheEnabled = true;
protected boolean callSettersOnNulls = false;
protected String logPrefix;
protected Class <? extends Log> logImpl;
protected LocalCacheScope localCacheScope = LocalCacheScope.SESSION;
protected JdbcType jdbcTypeForNull = JdbcType.OTHER;
protected Set<String> lazyLoadTriggerMethods = new HashSet<String>(Arrays.asList(new String[] { "equals", "clone", "hashCode", "toString" }));
protected Integer defaultStatementTimeout;
protected ExecutorType defaultExecutorType = ExecutorType.SIMPLE;
protected AutoMappingBehavior autoMappingBehavior = AutoMappingBehavior.PARTIAL;
protected Properties variables = new Properties();
protected ObjectFactory objectFactory = new DefaultObjectFactory();
protected ObjectWrapperFactory objectWrapperFactory = new DefaultObjectWrapperFactory();
protected MapperRegistry mapperRegistry = new MapperRegistry(this);
protected boolean lazyLoadingEnabled = false;
protected ProxyFactory proxyFactory;
protected String databaseId;
/**
* Configuration factory class.
* Used to create Configuration for loading deserialized unread properties.
*
* #see <a href='https://code.google.com/p/mybatis/issues/detail?id=300'>Issue 300</a> (google code)
*/
protected Class<?> configurationFactory;
protected final InterceptorChain interceptorChain = new InterceptorChain();
protected final TypeHandlerRegistry typeHandlerRegistry = new TypeHandlerRegistry();
protected final TypeAliasRegistry typeAliasRegistry = new TypeAliasRegistry();
protected final LanguageDriverRegistry languageRegistry = new LanguageDriverRegistry();
protected final Map<String, MappedStatement> mappedStatements = new StrictMap<MappedStatement>("Mapped Statements collection");
protected final Map<String, Cache> caches = new StrictMap<Cache>("Caches collection");
protected final Map<String, ResultMap> resultMaps = new StrictMap<ResultMap>("Result Maps collection");
protected final Map<String, ParameterMap> parameterMaps = new StrictMap<ParameterMap>("Parameter Maps collection");
protected final Map<String, KeyGenerator> keyGenerators = new StrictMap<KeyGenerator>("Key Generators collection");
protected final Set<String> loadedResources = new HashSet<String>();
protected final Map<String, XNode> sqlFragments = new StrictMap<XNode>("XML fragments parsed from previous mappers");
protected final Collection<XMLStatementBuilder> incompleteStatements = new LinkedList<XMLStatementBuilder>();
protected final Collection<CacheRefResolver> incompleteCacheRefs = new LinkedList<CacheRefResolver>();
protected final Collection<ResultMapResolver> incompleteResultMaps = new LinkedList<ResultMapResolver>();
protected final Collection<MethodResolver> incompleteMethods = new LinkedList<MethodResolver>();
...
Just don't worry about this.
After setup, MyBatis parsed mapper xml to a local variable(xml file read at first time).
org.apache.ibatis.session.Configuration.mappedStatements.
Now, you invoke mapper.add()/sqlSession.selectOne() or others, it will get this parameter/resultMap/resultType from mappedStatements firstly , won't read xml again.
Also, MyBatis cached the mapper proxy method. like this(create proxy method instance first time you invoke)
final String resource = "org/apache/ibatis/builder/MapperConfig.xml";
final Reader reader = Resources.getResourceAsReader(resource);
manager = SqlSessionManager.newInstance(reader);
AuthorMapper mapper = manager.getMapper(AuthorMapper.class);
Author expected = new Author(500, "cbegin", "******", "cbegin#somewhere.com", "Something...", null);
mapper.insertAuthor(expected);
(how to get mapper)
public class MapperRegistry {
public <T> T getMapper(Class<T> type, SqlSession sqlSession) {
final MapperProxyFactory<T> mapperProxyFactory = (MapperProxyFactory<T>) knownMappers.get(type);
if (mapperProxyFactory == null) {
throw new BindingException("Type " + type + " is not known to the MapperRegistry.");
}
try {
return mapperProxyFactory.newInstance(sqlSession);
} catch (Exception e) {
throw new BindingException("Error getting mapper instance. Cause: " + e, e);
}
}
}
And, u can use query cache in mybatis 3.x+. Following this
<configuration>
<settings>
<setting name="cacheEnabled" value="true" />
<settings>
</configuration>
Related
I am using Spring batch and have an ItemWriter as follows:
public class MyItemWriter implements ItemWriter<Fixing> {
private final FlatFileItemWriter<Fixing> writer;
private final FileSystemResource resource;
public MyItemWriter () {
this.writer = new FlatFileItemWriter<>();
this.resource = new FileSystemResource("target/output-teste.txt");
}
#Override
public void write(List<? extends Fixing> items) throws Exception {
this.writer.setResource(new FileSystemResource(resource.getFile()));
this.writer.setLineAggregator(new PassThroughLineAggregator<>());
this.writer.afterPropertiesSet();
this.writer.open(new ExecutionContext());
this.writer.write(items);
}
#AfterWrite
private void close() {
this.writer.close();
}
}
When I run my spring batch job, the items are written to file as:
Fixing{id='123456', source='TEST', startDate=null, endDate=null}
Fixing{id='1234567', source='TEST', startDate=null, endDate=null}
Fixing{id='1234568', source='TEST', startDate=null, endDate=null}
1/ How can I write just the data so that the values are comma separated and where it is null, it is not written. So the target file should look like this:
123456,TEST
1234567,TEST
1234568,TEST
2/ Secondly, I am having an issue where only when I exit spring boot application, I am able to see the file get created. What I would like is once it has processed all the items and written, the file to be available without closing the spring boot application.
There are multiple options to write the csv file. Regarding second question writer flush will solve the issue.
https://howtodoinjava.com/spring-batch/flatfileitemwriter-write-to-csv-file/
We prefer to use OpenCSV with spring batch as we are getting more speed and control on huge file example snippet is below
class DocumentWriter implements ItemWriter<BaseDTO>, Closeable {
private static final Logger LOG = LoggerFactory.getLogger(StatementWriter.class);
private ColumnPositionMappingStrategy<Statement> strategy ;
private static final String[] columns = new String[] { "csvcolumn1", "csvcolumn2", "csvcolumn3",
"csvcolumn4", "csvcolumn5", "csvcolumn6", "csvcolumn7"};
private BufferedWriter writer;
private StatefulBeanToCsv<Statement> beanToCsv;
public DocumentWriter() throws Exception {
strategy = new ColumnPositionMappingStrategy<Statement>();
strategy.setType(Statement.class);
strategy.setColumnMapping(columns);
filename = env.getProperty("globys.statement.cdf.path")+"-"+processCount+".dat";
File cdf = new File(filename);
if(cdf.exists()){
writer = Files.newBufferedWriter(Paths.get(filename), StandardCharsets.UTF_8,StandardOpenOption.APPEND);
}else{
writer = Files.newBufferedWriter(Paths.get(filename), StandardCharsets.UTF_8,StandardOpenOption.CREATE_NEW);
}
beanToCsv = new StatefulBeanToCsvBuilder<Statement>(writer).withQuotechar(CSVWriter.NO_QUOTE_CHARACTER)
.withMappingStrategy(strategy).withSeparator(',').build();
}
#Override
public void write(List<? extends BaseDTO> items) throws Exception {
List<Statement> settlementList = new ArrayList<Statement>();
for (int i = 0; i < items.size(); i++) {
BaseDTO baseDTO = items.get(i);
settlementList.addAll(baseDTO.getStatementList());
}
beanToCsv.write(settlementList);
writer.flush();
}
#PreDestroy
#Override
public void close() throws IOException {
writer.close();
}
}
Since you are using PassThroughLineAggregator which does item.toString() for writing the object, overriding the toString() function of classes extending Fixing.java should fix it.
1/ How can I write just the data so that the values are comma separated and where it is null, it is not written.
You need to provide a custom LineAggregator that filters out null fields.
2/ Secondly, I am having an issue where only when I exit spring boot application, I am able to see the file get created
This is probably because you are calling this.writer.open in the write method which is not correct. You need to make your item writer implement ItemStream and call this.writer.open and this this.writer.close respectively in ItemStream#open and ItemStream#close
I am new to Junit 5 . There are two functions in the class under test , The first function calls the second function and second function returns a value which is used in the first function for processing .
So I have created a mock for this class but not able to mock the second function call When I am testing the first function .
First function --exportOpportunityListing()
Second function -- entityToCsvReport()
public class OpportunityReportServiceImpl extends BaseService implements OpportunityReportService {
#Value("${nfs.mountPath}")
private String fileMountPath;
#Value("${take1.url.host}")
private String take1HostURL;
#Autowired
UsersRepository usersRepository;
#Autowired
MailUtil mailUtil;
#Autowired
OpportunityJDBCRepository ojdbc;
#Override
#Async
public void exportOpportunityListing(Map<String, Object> paramMap, List<OpportunityCriteria> lfvo,
String xRemoteUser) {
try {
List<OpportunityJDBCDTO> lo = ojdbc.getOppListWithoutPagination(paramMap, lfvo);
List<OpportunityReport> exportData = lo.parallelStream().map(this::entityToCsvReport)
.collect(Collectors.toList());
CsvCustomMappingStrategy<OpportunityReport> mappingStrategy = new CsvCustomMappingStrategy<>();
mappingStrategy.setType(OpportunityReport.class);
String dirPath = fileMountPath + REPORT_PATH;
File fileDir = new File(dirPath);
if (!fileDir.exists()) {
FileUtils.forceMkdir(fileDir);
}
String pathWithoutExtension = dirPath + "opportunity_data_"
+ LocalDateTime.now().format(DateTimeFormatter.ofPattern(YYYYMMDDHHMMSS));
File reportFile = new File(pathWithoutExtension + EXTENSION_CSV);
Writer writer = new PrintWriter(reportFile);
StatefulBeanToCsv<OpportunityReport> beanToCsv = new StatefulBeanToCsvBuilder<OpportunityReport>(writer)
.withMappingStrategy(mappingStrategy).build();
beanToCsv.write(exportData);
writer.close();
String zipFilePath = pathWithoutExtension + EXTENSION_ZIP;
ZipUtil.zip(reportFile, zipFilePath);
Users remoteUser = usersRepository.findByUsername(xRemoteUser)
.orElseThrow(() -> new Take1Exception(ErrorMessage.USER_NOT_FOUND_WITH_USERNAME, xRemoteUser));
Mail mail = Mail.builder().to(new String[] { remoteUser.getEmail() })
.model(MailModel.builder().name(remoteUser.getName())
.body("Please find attached the opportunity report you requested.").build())
.subject("Opportunity Report").attachments(Arrays.asList(new File(zipFilePath))).build();
mailUtil.sendMail(mail);
Files.delete(reportFile.toPath());
} catch (IOException | CsvDataTypeMismatchException | CsvRequiredFieldEmptyException e) {
throw new Take1Exception(ErrorMessage.INTERNAL_SERVER_EXCEPTION, e);
}
}
public OpportunityReport entityToCsvReport(OpportunityJDBCDTO o) {
OpportunityReport or = modelMapper.map(o, OpportunityReport.class);
or.setCurrency("USD");
or.setOnline(Boolean.TRUE.equals(o.getIsOnline()) ? "YES" : "NO");
return or;
}
}
Here is my JUnit Test case .
class OpportunityReportServiceImplTest {
#InjectMocks
OpportunityReportServiceImpl opportunityReportServiceImpl;
#Autowired
OpportunityReportServiceImpl ors;
#Mock
OpportunityJDBCRepository ojdbc;
#Mock
UsersRepository usersRepository;
#Mock
MailUtil mailUtil;
#Mock
ModelMapper mp;
String username = "anandabhishe";
String nfusername = "ananda";
Mail mail;
List<OpportunityJDBCDTO> lo = new ArrayList<OpportunityJDBCDTO>();
List<OpportunityReport> lor = new ArrayList<OpportunityReport>();
#BeforeEach
void setUp() throws Exception {
MockitoAnnotations.initMocks(this);
ReflectionTestUtils.setField(opportunityReportServiceImpl, "fileMountPath", ".");
ReflectionTestUtils.setField(opportunityReportServiceImpl, "take1HostURL", "");
lo.add(new OpportunityJDBCDTO());
lor.add(new OpportunityReport());
}
#Test
void testExportOpportunityListing() throws IOException {
OpportunityReport or = new OpportunityReport();
or.setCurrency("USD");
or.setOnline("Yes");
when(ojdbc.getOppListWithoutPagination(getParamMap(), getOppCriteria())).thenReturn(lo);
when(usersRepository.findByUsername(username)).thenReturn(Optional.of(getUser()));
doNothing().when(mailUtil).sendMail(mail);
// doNothing().when(opportunityReportServiceImpl).entityToCsvReport(oj);
when(opportunityReportServiceImpl.entityToCsvReport(getOpportunityJDBCDTO())).thenReturn(or);
opportunityReportServiceImpl.exportOpportunityListing(getParamMap(), getOppCriteria(), username);
assertTrue(true);
FileUtils.forceDelete(new File("." + REPORT_PATH));
}
private Map<String, Object> getParamMap() {
return new HashMap<String, Object>();
}
private List<OpportunityCriteria> getOppCriteria() {
List<OpportunityCriteria> loc = new ArrayList<>();
loc.add(new OpportunityCriteria());
return loc;
}
private OpportunityJDBCDTO getOpportunityJDBCDTO() {
OpportunityJDBCDTO oj = new OpportunityJDBCDTO();
oj.setIsOnline(true);
oj.setApplicationCount(2);
oj.setCost(200);
oj.setCountryCode("in");
oj.setCreationDate(LocalDateTime.now());
oj.setEndDate(LocalDate.now());
oj.setLocation("test");
oj.setOpportunityId(123);
oj.setOpportunityStatus("test");
oj.setOpportunityStatusId(1);
oj.setOpportunityTitle("test");
oj.setOpportunityType("test");
oj.setOpportunityTypeColor("test");
oj.setOpportunityTypeId(1);
oj.setPublishedAt(LocalDateTime.now());
oj.setPublishedBy("test");
oj.setPublishedByUserName("test");
oj.setRegistrationUrl("test");
oj.setStartDate(LocalDate.now());
oj.setSummary("test");
oj.setUserEmail("test");
oj.setUserFullName("test");
oj.setUserId(1);
oj.setUserName("test");
oj.setVendorName("test");
return oj;
}
private Users getUser() {
Users user = new Users();
return user;
}
}
I am getting Null Pointer Exception when the line in Test class is called :
when(opportunityReportServiceImpl.entityToCsvReport(getOpportunityJDBCDTO())).thenReturn(or);
I was missing mocking the modelmapper stub which is being used in second function , after I added that , the test passed .
OpportunityReport or = new OpportunityReport();
OpportunityJDBCDTO oj = new OpportunityJDBCDTO();
when(ojdbc.getOppListWithoutPagination(any(HashMap.class), anyList())).thenReturn(lo);
when(usersRepository.findByUsername(anyString())).thenReturn(Optional.of(getUser()));
doNothing().when(mailUtil).sendMail(mail);
doReturn(or).when(mp).map(oj, OpportunityReport.class);
opportunityReportServiceImpl.exportOpportunityListing(getParamMap(), getOppCriteria(), username);
assertTrue(true);
That's happening because opportunityReportServiceImpl is not a mock - it's the object that you're trying to test, but you're trying to stub a method of it as if it were a mock.
I would recommend that you don't try to stub the methods of the object that you're trying to test. But if you have to, you'll need to declare it as a #Spy. Then to stub it, you'll need the doReturn/when syntax instead of when/thenReturn. This might look like
doReturn(lo).when(ojdbc).getOppListWithoutPagination(getParamMap(), getOppCriteria());
I have a csv
id,name,description,price,date,name,address
1,SuperCsv,Write csv file,1234.56,28/03/2016,amar,jp nagar
I want to read it and store it to json file.
I have created two bean course(id,name,description,price,date) and person(name,address)
on reading by bean reader i'm not able to set the person address.
The (beautified) output is
Course [id=1,
name=SuperCsv,
description=Write csv file,
price=1234.56,
date=Mon Mar 28 00:00:00 IST 2016,
person=[
Person [name=amar, address=null],
Person [name=null, address=jpnagar]
]
]
I want the adress to set with name
My code:
public static void readCsv(String csvFileName) throws IOException {
ICsvBeanReader beanReader = null;
try {
beanReader = new CsvBeanReader(new FileReader(csvFileName), CsvPreference.STANDARD_PREFERENCE);
// the header elements are used to map the values to the bean (names must match)
final String[] header = beanReader.getHeader(true);
final CellProcessor[] processors = getProcessors();
final String[] fieldMapping = new String[header.length];
for (int i = 0; i < header.length; i++) {
if (i < 5) {
// normal mappings
fieldMapping[i] = header[i];
} else {
// attribute mappings
fieldMapping[i] = "addAttribute";
}}
ObjectMapper mapper=new ObjectMapper();
Course course;
List<Course> courseList=new ArrayList<Course>();
while ((course = beanReader.read(Course.class, fieldMapping, processors)) != null) {
// process course
System.out.println(course);
courseList.add(course);
}
private static CellProcessor[] getProcessors(){
final CellProcessor parsePerson = new CellProcessorAdaptor() {
public Object execute(Object value, CsvContext context) {
return new Person((String) value,null);
}
};
final CellProcessor parsePersonAddress = new CellProcessorAdaptor() {
public Object execute(Object value, CsvContext context) {
return new Person(null,(String) value);
}
};
return new CellProcessor[] {
new ParseInt(),
new NotNull(),
new Optional(),
new ParseDouble(),
new ParseDate("dd/MM/yyyy"),
new Optional(parsePerson),
new Optional(parsePersonAddress)
};
SuperCSV is the first parser I have seen that lets you create an object within an object.
for what you are wanting you can try Apache Commons CSV or openCSV (CSVToBean) to map but to do this you need to have the setters of the inner class (setName, setAddress) in the outer class so the CSVToBean to pick it up. That may or may not work.
What I normally tell people is to have a plain POJO that has all the fields in the csv - a data transfer object. Let the parser create that then use a utility/builder class convert the plain POJO into the nested POJO you want.
I am struggling with the creation of a SynonymFilter that I try to create programmatically. How are you supposed to tell the filter where the synonym list is?
I am using Hibernate Search, but I don't want to use the #AnalyzerDef annotation.
All I can do is pass a synonym map?
private class AllAnalyzer extends Analyzer {
private SynonymFilterFactory synonymFilterFactory = new SynonymFilterFactory();
public AllAnalyzer() {
ClassLoader classLoader = getClass().getClassLoader();
String filePath = classLoader.getResource("synonyms.txt").getFile();
HashMap<String, String> stringStringHashMap = new HashMap<String, String>();
stringStringHashMap.put("synonyms", filePath);
stringStringHashMap.put("format", "solr");
stringStringHashMap.put("ignoreCase", "false");
stringStringHashMap.put("expand", "true");
stringStringHashMap.put("luceneMatchVersion", Version.LUCENE_36.name());
synonymFilterFactory.init(stringStringHashMap);
}
#Override
public TokenStream tokenStream(String fieldName, Reader reader) {
TokenStream result = null;
result = new StandardTokenizer(Version.LUCENE_36, reader);
result = new StandardFilter(Version.LUCENE_36, result);
result = synonymFilterFactory.create(result);
return result;
}
}
Unable to get it to work. When I debug it says that the map is null and I get a NPE. What is wrong?
Yes, you need to pass a SynonymMap to the SynonymFilter.
Sounds like you want to populate it from a file, so you'll likely want to use SolrSynonymParser to generate it. Along the lines of:
SolrSynonymParser parser = new SolrSynonymParser(true, false, analyzer);
Reader synonymFileReader = new FileRader(new File(path));
parser.add(synonymFileReader);
SynonymMap map = parser.build(); // SolrSynonymParser extends SynonymMap.Builder
In vaadin 7, how does one lazily determine the file name when using FileDownloader?
final Button downloadButton = new Button("Download file");
FileDownloader downloader = new FileDownloader(new StreamResource(new StreamSource() {
#Override
public InputStream getStream () {
return new ByteArrayInputStream(expesiveCalculationOfContent());
}
}, "file.snub"));
downloader.extend(downloadButton);
In this code sample, clearly the filename
is rubbish
has to be known early on.
How can one lazily determine the filename of the downloaded file?
I do not know if it is dirty but this works: extending FileDownloader.handleConnectorRequest() to call the StreamResource.setFilename() prior to calling its super's method.
{
final Button downloadButton = new Button("Download file");
final StreamResource stream = new StreamResource(
new StreamSource() {
#Override
public InputStream getStream() {
return new ByteArrayInputStream("Hola".getBytes());
}
}, "badname.txt");
FileDownloader downloader = new FileDownloader(stream) {
#Override
public boolean handleConnectorRequest(VaadinRequest request,
VaadinResponse response, String path)
throws IOException {
stream.setFilename("better-name.txt");
return super
.handleConnectorRequest(request, response, path);
}
};
downloader.extend(downloadButton);
layout.addComponent(downloadButton);
}
This is the final solution I came up with:
/**
* This specializes {#link FileDownloader} in a way, such that both the file name and content can be determined
* on-demand, i.e. when the user has clicked the component.
*/
public class OnDemandFileDownloader extends FileDownloader {
/**
* Provide both the {#link StreamSource} and the filename in an on-demand way.
*/
public interface OnDemandStreamResource extends StreamSource {
String getFilename ();
}
private static final long serialVersionUID = 1L;
private final OnDemandStreamResource onDemandStreamResource;
public OnDemandFileDownloader (OnDemandStreamResource onDemandStreamResource) {
super(new StreamResource(onDemandStreamResource, ""));
this.onDemandStreamResource = checkNotNull(onDemandStreamResource,
"The given on-demand stream resource may never be null!");
}
#Override
public boolean handleConnectorRequest (VaadinRequest request, VaadinResponse response, String path)
throws IOException {
getResource().setFilename(onDemandStreamResource.getFilename());
return super.handleConnectorRequest(request, response, path);
}
private StreamResource getResource () {
return (StreamResource) this.getResource("dl");
}
}
If one assumes that lazily determining a file name means dynamically setting the filename irrespective of what the actual file system name may be, then the code below is what I'm using.
In the code below fileName points to the local file system file with a name that we want to change upon download. A usecase for this would be when one uploaded files to tmp with filenames containing some random characters that where not present in the original upload.
File file = new File(localFile);
final FileResource fileResource = new FileResource(file);
if (!file.exists()) {
throw new IllegalStateException();
}
final StreamResource stream = new StreamResource(
new StreamSource() {
#Override
public InputStream getStream() {
return fileResource.getStream().getStream();
}
}, "newname.txt");
FileDownloader fileDownloader = new FileDownloader(stream);
fileDownloader.extend(downloadButton);