We are trying to retrieve data from a Database and stream the result back to the REST client in a ZipOutputStream. The code is implemented in Camel
The data is quite large. Therefore loading all of it at once would cause memory problems.
We get the data in the sql camel component. Then split the result. Each splitted exchange creates a File stream (InputStream). All files should get zipped into 1 file and returned. How is this possible with Camel?
rest("/api/v1/request/{Id}/data")
.get().outType(StreamingOutput.class)
.route().routeId("getDataRoute")
.to("sql:SELECT * FROM Data WHERE ID_ID=:#id?outputType=StreamList")
.split(body()).parallelProcessing().stopOnException()
.bean(new FileStreamTranslator)
//Generate a filestream per split. Zip the files and return the ZipStream. how can this be achieved?
Related
I am trying to write a small spring boot application, which needs to do the following:
--Create and Save:
-Query things from database (List of result sets, to be converted to pojo)
-Convert that pojo to csv (without the actual file)
-Save that csv to database as blob/bytea (postgres).
--Retrieve:
-Get the bytea from db, send that as part of json response from a GET endpoint
I have managed to create the csv via OpenCSV and using StatefulBeanToCsv - but since this uses Printwriter, I am worried about having memory issues when processing large files.
Any alternatives so I can do above requirements?
I have a Java program (a war) that runs out of memory when manipulating a big XML file.
The program is a REST API that returns the manipulated XML via a REST Controller.
First, the program gets an XML file from a remote URL.
Then it replaces the values of id attributes.
Finally, it returns the new XML to the caller via the API controller.
What I get from the remote URL is a byte[] body with XML data.
Then, I convert it to a String.
Next, I do a regexp search-replace on the whole string.
Then I convert it back to a byte[].
I'm guessing that the XML now is in memory 3 times (the incoming bytes, the string and the outgoing bytes).
I'm looking for ways to improve this.
I have no local copies on the filesystem btw.
You can delete the incoming bytes from memory after converting the bytes to String:
byte[] bytes = bytesFromURL;
String xml = new String(bytes);
{...manipulate xml}
bytes = null;
System.gc();
bytes = xml.getBytes();
I am faced with a problem. I have a mp4 file which has been run through mp4box. I run it with the interleaved option to generated metadata for the file. I can dump an XML file of the metadata which contains data like
<CompositionOffsetEntry CompositionOffset="1127" SampleCount="1"/>
<SampleSizeEntry Size="2581"/>
<ChunkEntry offset="17065364"/>
<TimeToSampleEntry SampleDelta="1024" SampleCount="6"/>
<SampleToChunkEntry FirstChunk="2525" SamplesPerChunk="19" SampleDescriptionIndex="1"/>
<SampleSizeEntry Size="301"/>
etc...
What I need to know is how to read and understand this data in order to map seconds, key-frames and bytes. For example, I want to be able to tell that second 100 maps keyframe 500 maps to byte 30520 (From the beginning of the .mp4 file).
I am integrating data between two systems using Apache Camel. I want the resulting xml to be written to an xml file. I want to base the name of that file on some data which is unknown when the integration chain starts.
When I have done the first enrich step the data necessary is in the Exchange object.
So the question is how can I get data from the exchange.getIn().getBody() method outside of the process chain in order to generate a desirable filename for my output file and as a final step, write the xml to this file? Or is there some other way to accomplish this?
Here is my current Process chain from the routebuilders configuration method:
from("test_main", "jetty:server")
.process(new PiProgramCommonProcessor())
.enrich("piProgrammeEnricher", new PiProgrammeEnricher())
// after this step I have the data available in exchange.in.body
.to(freeMarkerXMLGenerator)
.to(xmlFileDestination)
.end();
best regards
RythmiC
The file component takes the file name from a header (if present). So you can just add a header to your message with the desired file name.
The header should use the key "CamelFileName" which is also defined from Exchange.FILE_NAME.
See more details at: http://camel.apache.org/file2
I have a JSP in which I want to:
make an HTTP request;
get back the XML response;
transform the response using XSL; and
export the transformed data to excel.
But i do not get the entire data in a single HTTP call. Also I have to use two stylesheets for exporting to Excel. The first is for the first page of records, and the other is for all the other pages.
How can one do this in Java?
Thanks
Or use Apache POI, its a powerful API to read and create MS Office documents in Java.
Could you parse the XML into Java objects, then when you have the complete object list simply write the Excel file using JExcelAPI? I've used this for Excel files and it's very simple & useful.