Develop Reference

Import and Reading a CsvFile with dedicated service to a existing repo with its own service

Trying to add some enhancements to this app,
private void parseCsv(CsvMapReader csvMapReader) throws IOException {
String[] header = csvMapReader.getHeader(true);
List<String> headers = Arrays.asList(header);
verifySourceColumn(headers);
verifyPovColumn(headers);
final CellProcessor[] processors = getProcessors(headers);
Map<String, Object> csvImportMap = null;
while ((csvImportMap = csvMapReader.read(header, processors)) != null) {
CsvImportDTO csvImportDto = new CsvImportDTO(csvImportMap);
if ( activationTypeP(csvImportDto) ){
AipRolloutVO aipRolloutVO = new AipRolloutVO(csvImportDto.getSource(),
csvImportDto.getPov(),
csvImportDto.getActivationType(),
csvImportDto.getActivationDate(),
csvImportDto.getDeactivationDate(),
csvImportDto.getMssValue());
aipRolloutRepository.updateAipRollout(aipRolloutVO.getDc(),
aipRolloutVO.getPov(),
aipRolloutVO.getActivationType(),
aipRolloutVO.getMssValue());
}
}
}
When it goes to the repo I get:
cannot find local variable 'csvImportMap'
5 times and then:
((CsvParserService)this).aipRolloutService = inconvertiible types; cannont cast 'org.spring.....
my controller method
#PostMapping(value = "/updatecsv", produces = MediaType.APPLICATION_JSON_VALUE)
#ResponseBody
public ResponseEntity<?> processCsv( #RequestParam("csvFile") MultipartFile csvFile) throws IOException {
if (csvFile.isEmpty()) return new ResponseEntity(
responceJson("please select a file!"),
HttpStatus.NO_CONTENT
);
csvParserService.parseCsvFile(csvFile);
return new ResponseEntity(
responceJson("Successfully uploaded - " + csvFile.getOriginalFilename()),
new HttpHeaders(),
HttpStatus.CREATED
);
}
repo im trying to reuse to update these values
public int updateAipRollout(String dc, String pov, String activationType, int mssValue) {
String query = some query
logger.debug("Rollout update query: " + query);
int num =jdbcTemplate.update(query);
return num;
}
Do I need to autowire the other service class that this repo is in and then call that service? But when I did that it also didn't fix the error....

Spring WebClient corrupts binary data

I'm sending files containing binary data from service A to service B. When the number of files is relatively small (let's say 5) everything works well. However, when I try to send more files (let's say several hundred) it sometimes fails. I tried to check what is happening with this binary data, and it looks like WebClient corrupts it in some way (weird padding appears at the end).
I created a minimal reproducible example to illustrate this issue.
Endpoint in service B (consuming binary files):
#RestController
class FilesController {
#PostMapping(value = "/files")
Mono<List<String>> uploadFiles(#RequestBody Flux<Part> parts) {
return parts
.filter(FilePart.class::isInstance)
.map(FilePart.class::cast)
.flatMap(part -> DataBufferUtils.join(part.content())
.map(buffer -> {
byte[] data = new byte[buffer.readableByteCount()];
buffer.read(data);
DataBufferUtils.release(buffer);
return Base64.getEncoder().encodeToString(data);
})
)
.collectList();
}
}
Tests illustrating how the service A sends data:
public class BinaryUploadTest {
private final CopyOnWriteArrayList<String> sentBytes = new CopyOnWriteArrayList<>();
#BeforeEach
void before() {
sentBytes.clear();
}
/**
* this test passes all the time
*/
#Test
void shouldUpload5Files() {
// given
MultiValueMap<String, HttpEntity<?>> body = buildResources(5);
// when
List<String> receivedBytes = sendPostRequest(body);
// then
assertEquals(sentBytes, receivedBytes);
}
/**
* this test fails most of the time
*/
#Test
void shouldUpload1000Files() {
// given
MultiValueMap<String, HttpEntity<?>> body = buildResources(1000);
// when
List<String> receivedBytes = sendPostRequest(body);
// then
assertEquals(sentBytes, receivedBytes);
}
private List<String> sendPostRequest(MultiValueMap<String, HttpEntity<?>> body) {
return WebClient.builder().build().post()
.uri("http://localhost:8080/files")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(BodyInserters.fromMultipartData(body))
.retrieve()
.bodyToMono(new ParameterizedTypeReference<List<String>>() {
})
.block();
}
private MultiValueMap<String, HttpEntity<?>> buildResources(int numberOfResources) {
MultipartBodyBuilder builder = new MultipartBodyBuilder();
for (int i = 0; i < numberOfResources; i++) {
builder.part("item-" + i, buildResource(i));
}
return builder.build();
}
private ByteArrayResource buildResource(int index) {
byte[] bytes = randomBytes();
sentBytes.add(Base64.getEncoder().encodeToString(bytes)); // keeps track of what has been sent
return new ByteArrayResource(bytes) {
#Override
public String getFilename() {
return "filename-" + index;
}
};
}
private byte[] randomBytes() {
byte[] bytes = new byte[ThreadLocalRandom.current().nextInt(16, 32)];
ThreadLocalRandom.current().nextBytes(bytes);
return bytes;
}
}
What could be the reason for this data corruption?

It turned out to be a bug in the Spring Framework (in the MultipartParser class to be more precise). I have created a GitHub issue which will be fixed in the next version (5.3.16). The bug is fixed by this commit.

Can't manage to test rest template helper class

I'm trying for more than an hour to test this class. It went so ugly of stubbing the whole components of the method etc. I'd love some advice how to make a better test or refactor the class to make it way easier to test. I could not figure out a way yet.
Class to Test
#Slf4j
public final class HistoryRestService {
static RestTemplate restTemplate = new RestTemplate();
public static Optional<List<History>> findLatestHistories() {
String url = buildUrl();
ResponseEntity<History[]> responseEntity = null;
try {
responseEntity = restTemplate.getForEntity(url, History[].class);
} catch (ResourceAccessException e) {
log.warn("No connection to History persistence. Please check if the history persistence started up properly");
return Optional.empty();
}
History[] histories = responseEntity.getBody();
return Optional.of(Arrays.asList(histories));
}
private static String buildUrl() {
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("http://");
stringBuilder.append("localhost");
stringBuilder.append(":8081");
stringBuilder.append("/history/get");
return stringBuilder.toString();
}
// For Testing
static void setRestTemplate(RestTemplate restTemplate) {
HistoryRestService.restTemplate = restTemplate;
}
}
Spock Test which fails
class HistoryRestServiceTest extends Specification {
def "test findLatestHistories"() {
given:
History mockedHistory = Mock()
HistoryRestService uut = new HistoryRestService()
History[] expected = [mockedHistory]
RestTemplate mockedRestTemplate = Stub()
ResponseEntity<History> mockedResponseEntity = Stub()
mockedResponseEntity.getBody() >> expected
mockedRestTemplate.getForEntity(_) >> mockedResponseEntity
uut.setRestTemplate(mockedRestTemplate)
when:
def actual = uut.findLatestHistories()
then:
actual.get() == expected
}
}

I'd suggest using real depedency-injection (spring/guice/cdi) instead of static variables.
Furthermore, you should think about what you want to test, is it the correct request and parsing of the network call, then write an integration test using something like mockserver or wiremock to have the whole stack. Or, if you are just concerned with the result handling, then you could move the code that interacts with RestTemplate into a separate method and use partial mocking to mock this method. I'd suggest to use the real integration test, but for the sake of an example this should work, but I didn't verify the code.
#Slf4j
public class HistoryRestService {
private final RestTemplate restTemplate;
public HistoryRestService() {
restTemplate = new RestTemplate();
}
public HistoryRestService(RestTemplate restTemplate) {
this.restTemplate = restTemplate;
}
public Optional<List<History>> findLatestHistories() {
try {
return Optional.of(Arrays.asList(getLatestHistories(buildUrl())));
} catch (ResourceAccessException e) {
log.warn("No connection to History persistence. Please check if the history persistence started up properly");
return Optional.empty();
}
}
History[] getLatestHistories(String url) throws {
ResponseEntity<History[]> responseEntity = null;
responseEntity = restTemplate.getForEntity(url, History[].class);
return responseEntity.getBody()
}
private String buildUrl() {
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("http://");
stringBuilder.append("localhost");
stringBuilder.append(":8081");
stringBuilder.append("/history/get");
return stringBuilder.toString();
}
}
class HistoryRestServiceTest extends Specification {
#Subject
HistoryRestService uut = Spy()
def "test findLatestHistories"() {
given:
History[] expected = [mockedHistory]
when:
def actual = uut.findLatestHistories()
then:
actual.get() == expected
1 * uut.getLatestHistories(_ as String) >> expected
}
def "test findLatestHistories returns empty on exceptions"() {
given:
History[] expected = [mockedHistory]
when:
def actual = uut.findLatestHistories()
then:
!actual.present
1 * uut.getLatestHistories(_ as String) >> {throw new ResourceAccessException()}
}
}

Notification for HTTP POST

I am new to Spring framework and have tried creating an api for others to use.
This is in my controller class
POST method
// -------------------Create a Report-------------------------------------------
#RequestMapping(value = "/report/", method = RequestMethod.POST)
public ResponseEntity<?> createReport(#RequestBody Report report, UriComponentsBuilder ucBuilder) {
logger.info("Creating Report : {}", report);
if (reportRepository.isReportExist(report)) {
logger.error("Unable to create. A report with name {} already exist", report.getCrisisID());
return new ResponseEntity(new CustomErrorType("Unable to create. A Report with crisisID " +
report.getCrisisID() + " already exist."),HttpStatus.CONFLICT);
}
reportRepository.saveReport(report);
HttpHeaders headers = new HttpHeaders();
headers.setLocation(ucBuilder.path("/api/report/{crisisID}").buildAndExpand(report.getCrisisID()).toUri());
return new ResponseEntity<String>(headers, HttpStatus.CREATED);
}
GET Method
// -------------------Retrieve All Reports---------------------------------------------
#RequestMapping(value = "/report/", method = RequestMethod.GET)
public ResponseEntity<List<Report>> listAllReports() {
List<Report> reports = reportRepository.findAllReports();
if (reports.isEmpty()) {
return new ResponseEntity(HttpStatus.NO_CONTENT);
// You many decide to return HttpStatus.NOT_FOUND
}
return new ResponseEntity<List<Report>>(reports, HttpStatus.OK);
}
Below is the an example code for others to create the report object and send it to my api.
// POST
private static void createReport() {
System.out.println("Testing create Report API----------");
RestTemplate restTemplate = new RestTemplate();
Report report = new Report(20, "General", 4, "AA1",10000,"crisis details", "1 hour", "COA1", "COA1");
URI uri = restTemplate.postForLocation(REST_SERVICE_URI + "/report/", report,
Report.class);System.out.println("Location : "+uri.toASCIIString());
}
I am wondering if the following is possible or if there is any way to approach this situation:
When someone creates a report and sends it to me via POST URL, my POST method will be able to automatically detect that a new report entry is created and then sends a notification in my HTML/JSP page (such as a pop up window).
Update
RestApiController.java
#RestController
#RequestMapping("/api")
public class RestApiController {
public static final Logger logger = LoggerFactory.getLogger(RestApiController.class);
#Autowired
ReportRepository reportRepository; //Service which will do all data retrieval/manipulation work
// -------------------Create a Report-------------------------------------------
#RequestMapping(value = "/report/", method = RequestMethod.POST)
#SendTo("/channel/publicreport")
public ResponseEntity<?> createReport(#RequestBody Report report, UriComponentsBuilder ucBuilder) {
logger.info("Creating Report : {}", report);
if (reportRepository.isReportExist(report)) {
logger.error("Unable to create. A report with name {} already exist", report.getCrisisID());
return new ResponseEntity(new CustomErrorType("Unable to create. A Report with crisisID " +
report.getCrisisID() + " already exist."),HttpStatus.CONFLICT);
}
reportRepository.saveReport(report);
HttpHeaders headers = new HttpHeaders();
headers.setLocation(ucBuilder.path("/api/report/{crisisID}").buildAndExpand(report.getCrisisID()).toUri());
return new ResponseEntity<String>(headers, HttpStatus.CREATED);
}
}
WebSocketConfig.java
#Configuration
#EnableWebSocketMessageBroker
public class WebSocketConfig extends AbstractWebSocketMessageBrokerConfigurer {
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/chatservice");
}
#Override
public void configureMessageBroker(MessageBrokerRegistry registry) {
registry.setApplicationDestinationPrefixes("/app");
registry.enableSimpleBroker("/channel");
}
}
main.js
'use strict';
var stompClient = null;
var username = null;
function connectReport(event) {
username = "myname";
if(username) {
//var socket = new SockJS('/ws');
//stompClient = Stomp.over(socket);
stompClient = Stomp.client('ws://localhost:8080/chatservice');
stompClient.connect({}, onConnectedReport, onError);
}
event.preventDefault();
}
function onConnectedReport() {
// Subscribe to the Public Channel
stompClient.subscribe('/channel/publicreport',onReportMessageReceived);
}
function onError(error) {
connectingElement.textContent = 'Could not connect to WebSocket server. Please refresh this page to try again!';
connectingElement.style.color = 'red';
}
function onReportMessageReceived(payload) {
//Code for pop up window
}
window.addEventListener("load", connectReport, true)

In general taks like this is delegated to client. I.e. client polls the server for the changes it is interested in and once detected the client reacts accordingly. The oldest way (which is now absolete) was to add a tag into HTML that forces the entire page to refresh with a certain frequency which means the page sends a request to the server and receives an updated page to display. IN our days there are all kinds of frameworks that do partial updates to the page. One of the first ones was Ajax, then DHTML and so on and so forth. I am not a client side progrqammer. But as far as the concept goes such task is usually given to client

Parsing GET JSON reply without keys to object in Spring

I am currently consuming content in the style of
"potter",["potter harry","potter","potter hallows 2","potter collection","potter hallows 1","potter dvd box","potter 7","potter hallows","potter blue ray","potter feniks"],[{"xcats":[{"name":"dvd_all"},{"name":"books_nl"}]},{},{},{},{},{},{},{},{},{}],[]]
Using the following code in Spring
RestTemplate restTemplate = new RestTemplate();
String information = restTemplate.getForObject(URL, String.class);
//further parsing of information, using String utilities
Obviously this is not the way to go, because I should be able to automatically parse it somehow. I will also only need the content of the second element as well (the array, from potter harry to potter feniks).
What is the best way to parse a GET response like that, when its json contents aren't name-valued?

#Test
public void testJson(){
RestTemplate template = new RestTemplate();
ResponseEntity<ArrayNode> entity = template.
exchange("https://api.myjson.com/bins/2rl7m", HttpMethod.GET, null, new ParameterizedTypeReference<ArrayNode>() {
});
ArrayNode body = entity.getBody();
body.get(1).forEach(m->{
System.out.println(m.asText());});
}
But my advice is if you can change the response type not to be json array with mixed value types in it will be better

The following helper class enabled me to easily parse the response and get the list I needed
public class JSONHelper {
private static final JsonFactory factory = new JsonFactory();
private static final ObjectMapper mapper = new ObjectMapper(factory);
public static List<String> getListOnPosition(int i, String inputWithFullList) throws JsonProcessingException, IOException {
List<String> result = new ArrayList<String>();
JsonNode rootNode = mapper.readTree(inputWithFullList);
ArrayNode node = (ArrayNode) rootNode.get(i);
if (!node.isArray()) {
result.add(node.asText());
} else {
for (final JsonNode subNode : node) {
result.add(subNode.asText());
}
}
return result;
}
}
Some JUnit tests for this scenario
public class JSONHelperTest {
#Test
public void parseListOnPositionFullListTest() throws JsonProcessingException, IOException {
String inputWithFullList = "[\"a\",[\"b\", \"c\", \"d\"],[],[]]";
List<String> result = JSONHelper.getListOnPosition(1, inputWithFullList);
assertEquals(3, result.size());
assertEquals(Arrays.asList("b", "c", "d"), result);
}
#Test
public void parseListOnPositionOneElementListTest() throws JsonProcessingException, IOException {
String inputWithFullList = "[\"a\",[\"b\"],[],[]]";
List<String> result = JSONHelper.getListOnPosition(1, inputWithFullList);
assertEquals(1, result.size());
assertEquals(Arrays.asList("b"), result);
}
#Test
public void parseListOnPositionEmptyListTest() throws JsonProcessingException, IOException {
String inputWithFullList = "[\"a\",[],[],[]]";
assertTrue(JSONHelper.getListOnPosition(1, inputWithFullList).isEmpty());
}
}