I have been given a bit of a strange task, there are around 1500-2000 jpeg images all of around 1-50kb in size. They are currently stored in a simple database I made with Postgres. It's been a long time since I used Matlab and Postgres heavily so any help or suggestions is really appreciated!
I need to get the images that are stored in the database, out of the database into Java. The last step is retrieve the image from Java into Matlab so that the image is stored in the same way in which the imread function works in Matlab. The imread function reads an image in and creates a n by m by 3 matrix array of uint8 values which denote the pixel intensities of RGB.
Atm I have got the image in and out of the database in Java, currently storing the image in a bytea column data type. Is the best data type to use?
How can I get the data back out from the database, so that it is either the constructed jpeg image which I put in or is in the requested matrix array format?
Currently I do not understand the retrieved data. It is in a byte array of around 70,000 elements containing values between -128 to 128. Help!?!
Note: The database toolkit is unavailable to me
ANOTHER UPDATE: I have solved the problem related to the post regarding'UTF-8' encoding error.
If anyone stumbles upon this page, any answer posted will be tried as soon as I can! I really do appreciate your thoughts and answers. Thanks again.
When you say you have the image in a bytea column, how exactly is it stored? Is it storing the bytes of the JPEG file's contents, or an array of RGB pixel values, or something else? "Bytea" is just a binary string, which can store data in pretty much any format.
I'm assuming it's the JPEG contents. In that case what you can do is retrieve the jpeg contents via Java, save them to a temporary file, and call imread() on the temp file.
Those [-128,127] values are values of signed bytes Java. Even without the Database Toolbox, you can call regular JDBC or other Java code that uses it. The Java method you used to get those values - call it from Matlab (with your JAR on the classpath), and it should return that array as an int8 array, or something you can convert to one.
Given that in a Matlab variable named "bytes", you can write it out to a temp file with something like this.
file = [tempname() '.jpg'];
fid = fopen(file, 'wb');
fwrite(fid, bytes, 'int8');
fclose(fid);
By specifying 'int8' precision, I think you can skip the step of converting them to unsigned bytes, which is a more common convention. Writing int8s as 'int8' or uint8s as 'uint8' will produce the same file. If you do need to convert them to unsigned, use Matlab's typecast() function.
unsigned_bytes = typecast(bytes, 'uint8');
At that point you can call imread on the temp file and then delete it.
img = imread(file);
delete(file);
Problem solved :-)
I have managed to get the bytes stored within the bytea column within the database into a byte array. Then through creating a temp file ( using ByteArrayInputStream and Reader object to form a BufferedImage object which I write to a file) send this back into Matlab in an array.
Then process the data I retrieved and read from the temporary file in Matlab. Once the data is within Matlab all the temp files are deleted.
The code to process the ResultSet to create a temp image from a byte array received from the databases bytea column is shown below:
private static void processImageResultSet(ResultSet rs) throws SQLException, FileNotFoundException, IOException{
int i = 0; //used as a count and to name various temp files
while(rs.next()){ //loop through result sets
byte[] b = rs.getBytes(1); //the bytea column result
String location = getFileName(rs.getString(2)); //the name of the jpg file
ByteArrayInputStream bis = new ByteArrayInputStream(b); //creates stream storing byts
//To make individual names of temporary files unique the current time and date is stored
SimpleDateFormat df = new SimpleDateFormat("'Date 'yyyy-MM-dd HH'H'-mm'M'-ss'secs'-SS'ms'"); //formats date string
Calendar cal = Calendar.getInstance(); //gets instance of calendar time
String fileDate = df.format(cal.getTime()); //gets the time and date as a String
Iterator<?> readers = ImageIO.getImageReadersByFormatName("jpg"); //creates a reader object, that will read jpg codec compression format
Object source = bis; //object to store stream of bytes from database
ImageReader reader = (ImageReader) readers.next();
ImageInputStream iis = ImageIO.createImageInputStream(source); //creates image input stream from object source which stores byte stream
reader.setInput(iis, true); //sets the reader object to read image input stream
ImageReadParam param = reader.getDefaultReadParam();
Image image = reader.read(0, param);
BufferedImage bufferedImage = new BufferedImage(image.getWidth(null), image.getHeight(null), BufferedImage.TYPE_INT_RGB); //creates buffered image
Graphics2D g2 = bufferedImage.createGraphics();
g2.drawImage(image, null, null);
File imageFile = new File(location + " " + fileDate + " " + i + ".jpg"); //creates image file
ImageIO.write(bufferedImage, "jpg", imageFile); //writes buffered image object to created file
i++; //counts number of results from query within the ResultSet
}
}
Do you have access to the Database Toolbox in MATLAB? If so, you should be able to directly connect to a PostgreSQL database using the DATABASE function and then import and export data using the FETCH function or the QUERYBUILDER GUI. This may be easier than first going through Java.
Related
Uploading an image to hbase using Java program, after retrieving the image I found there is difference in file size eventually increased and most of Exif and Meta data loss
(GPS location data, camera details, etc..)
Code :
public ArrayList<Object> uploadImagesToHbase(MultipartFile uploadedFileRef){
byte[] bytes =uploadedFileRef.getBytes();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ImageIO.write(image, "jpg", outputStream);
HBaseAdmin admin = new HBaseAdmin(configuration);
HTable table = new HTable(configuration, "sample");
Put image = new Put(Bytes.toBytes("1"));
image.add(Bytes.toBytes("DataColumn"), Bytes.toBytes(DataQualifier), bytes);
table.put(image);
How to store and retrieve a Image with out any change / loss?
Please try using SerializationUtils from Apache Commons Lang.
Below are methods
static Object clone(Serializable object) //Deep clone an Object using serialization.
static Object deserialize(byte[] objectData) //Deserializes a single Object from an array of bytes.
static Object deserialize(InputStream inputStream) //Deserializes an Object from the specified stream.
static byte[] serialize(Serializable obj) //Serializes an Object to a byte array for storage/serialization.
static void serialize(Serializable obj, OutputStream outputStream) //Serializes an Object to the specified stream.
While storing in to hbase you can store byte[] which was returned from serialize.
While getting the Object you can type cast to corresponding object for ex: File object and can get it back.
Most likely you are just over-complicating things. :-)
The reason why you are losing the Exif and other metadata, is that the ImageIO convenience methods ImageIO.read(...) and ImageIO.write(...) does not preserve metadata. The good news is, they are not needed.
As you seem to already have the image data from the MultipartFile, you should simply store that data (the byte array) in the database, and you will store exactly what the user uploaded. No difference in file size, and metadata will be untouched.
Your code above doesn't compile for me, and I'm no HBase expert, so I just leave that out (as you have already been able to store an image, to see the size/quality difference and metadata loss, I assume you know how to do that :-) ). But here's the basics:
public ArrayList<Object> uploadImagesToHbase(MultipartFile uploadedFileRef) {
byte[] bytes = uploadedFileRef.getBytes();
// Store the above "bytes" byte array in HBase *as is* (no ImageIO)
}
I am working on a small POC (which will be integrated into a bigger application) which consists of
Problem Context
reading an image in a simple java application program,
Convert image into a byte array byte[] imageEncodedBytes = baos.toByteArray()
storing it into a remote DB2 database (technology used was not imp, so I am using plain jdbc for now)
Reading it back from the DB and
converting it back to ensure that the re-creation of the image works. ( I am able to open the new re-created image in any image viewer)
Issues
The issues occur at step 5.
I read the image using a select query into a Result Set.
use the rs.getBlob("ColumnName") to get the blob value.
Fetch the byte array from the blob value using byte[] decodedArray = myBlob.getBytes(1, (int)myBlob.length())
Create the image from the obtained byte array.
At Step 3 the byte array decodedArray obtained from the blob differs from the byte array 'imageEncodedBytes' that I get when I read the image.
As a consequence, the following code to create the image from the byte array decodedArray fails.
ByteArrayInputStream bais = new ByteArrayInputStream(decodedArray);
//Writing to image
BufferedImage imag=ImageIO.read(bais); // Line of failure. No registered provider able to read bais
ImageIO.write(imag, "jpg", new File(dirName,"snap.jpg"));
References and Other data for issue investigation
I have referred the following links for verification
1. Inserting image in DB2
2. This Link here offers insight, but yet I was not able to determine, how to register the ImageReader.
4. When inserting the image to DB2 I am using - the following query
Statement st = conn.createStatement();
st.executeUpdate("INSERT INTO PHOTO (ID,PHOTO_NM,PHOTO_IM, THMBNL_IM) " + "VALUES (1,'blob("+bl+")',blob('"+bl+"')")
As an alternative to fetching blob value from the result set I have also used binaryStream = rs.getBinaryStream("PHOTO_IM") to get the binary stream and then get byte array from the binary stream. even in this case, the decodedArray is different from imageEncodedBytes
Please assist, I may be missing something extremely trivial here, but I am not able to figure out what. Any help/pointers will be greatly helpful. Thanks in advance.
The resolution is sort of a workaround that I have worked on.
I have used the jdbcTemplates to resolve the issue. The lobHandler Object of the jdbc template provides an easy way to manage the blobs.
Steps to resolve using the Spring Lob Handler
) Created a Data Source
) configured the Jdbc template to use the data source
) use the lobHandler code to execute the insert query
Code below
jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.execute("INSERT INTO PHOTO (PHOTO_IM) VALUES (?)",
new AbstractLobCreatingPreparedStatementCallback(lobHandler) {
protected void setValues(PreparedStatement ps, LobCreator lobCreator) {
try {
lobCreator.setBlobAsBinaryStream(ps, 1, bean.getImageOrig(), bean.getImageLength());
} catch (java.sql.SQLException e) {
e.printStackTrace();
}
}
}
);
references
1. ) Stack Overflow Link - Spring JDBC Template Insert Blob
2. ) Stack Overflow Link - Close InputStream
3. ) Spring Doc for LobHandler
I am working around my project and got some Problem, I searched for it , but couldn't find meaning full learning resource, What I need is to Store Image in SQL Server Database using my Java program , and will need that Back to Retrieve, images are not of larger size they are ranging between 30 and 50 K,
I can load images from my Disk by using getImage() method in toolKit
Image imm = Toolkit.getDefaultToolkit().getImage("URl");
, but i don't know How to convert that image to Binary Format and to store in Database, and then retrieve Back From Database.
I want to Store that in VarBinary s by looking around several sites i found that the image type in SQL Server is soon going to Retire.
While it's not a good idea to store very large binary objects in the database, the Microsoft research paper "To BLOB or Not To BLOB: Large Object Storage in a Database or a Filesystem" indicates it's an efficient approach if object sizes are less than 256K. So it sounds like you've hit the sweet spot for database storage with 30K images.
You can load your image as a buffered image from a URL using:
BufferedImage imm = ImageIO.read(url);
Then convert it to a byte array with:
byte[] immAsBytes =
ByteArrayOutputStream baos = new ByteArrayOutputStream();
//use another encoding if JPG is innappropriate for you
ImageIO.write(imm, "jpg", baos );
baos.flush();
byte[] immAsBytes = baos.toByteArray();
baos.close();
Then store the byte array as recommended by this article as a VARBINARY in the database. In your JDBC code, you should wrap the byte array in a ByteArrayInputStream and set the PreparedStatement parameter as a BinaryStream:
PreparedStatement pstmt = commection.prepareStatement("INSERT INTO IMAGES (image) VALUES(?)");
ByteArrayInputStream bais = new ByteArrayInputStream(immAsBytes);
pstmt.setBinaryStream(1, bais, immAsBytes.length);
pstmt.executeUpdate();
pstmt.close();
To get the image out of the database, use ResultStatement.getBlob():
Blob immAsBlob = rs.getBlob();
byte[] immAsBytes = immAsBlob.getBytes(1, (int)immAsBlob.length()));
Finally, convert the byte array to a BufferedImage and do something with it:
InputStream in = new ByteArrayInputStream(immAsBytes);
BufferedImage imgFromDb = ImageIO.read(in);
It is generally not a very good idea to put into the database large resources like images, unless you really need ACID compliance when working with them. Database dumps and time required to produce them grow tremendously and you will need to use the more complex and fragile incremental backups.
Generally it may be more efficient to store the images on the filesystem, and only put the image path to the database after you are sure the image resource has been successfully created.
I've stored a Blob (PNG file) into the database like so:
File file = new File( "image.png" );
FileInputStream fis = new FileInputStream( file );
statement.setBinaryStream( 1, fis, (int) file.length() );
Currently I use this process to get the Blob image from the DB and convert it back into an image for use:
Blob blob = results.getBlob( 1 );
ImageIcon imageIcon = new ImageIcon( blob.getBytes( 1L, (int) blob.length() ) );
However, i need a method to put the image back into the database (after i've altered it) from the ImageIcon without creating a file, storing it to the disk then reading it back in with the FileInputSteam.
edit for clarity
Well, say i've got that image stored inside a Label as an ImageIcon. The only way i know how to put that into the database is to read from a FileInputStream, but that would involve pointlessly making a file of the image. So how would i read the Image from ImageIcon back out as a BinaryStream or Blob back to the database?
I'd try something like this
PixelGrabber pg = new PixelGrabber(imageIcon.getImage(),0,0, w,h,true);
pg.grabPixels();
// You may need to wait here until all pixels are copied (see ImageObserver)
Object buffer = pg.getPixels();
See http://download.oracle.com/javase/1.4.2/docs/api/java/awt/image/PixelGrabber.html
You may need to tweak it appropriately and specify correct w and h.
In the Blob API:
setBinaryStream(long pos) returns an OutputStream
"Retrieves a stream that can be used to write to the BLOB value that this Blob object represents." So you can write your bits directly to that stream.
I'm not quite sure how to get the pixels from your modified ImageIcon. If you can cast it to a BufferedImage, that class has a getRGB method.
I am struggling with the transfer of a simple jpeg file inside an ID3v2 tag from c++ over a TCP socket to java (Android). The library "taglib" offers to extract this file and I am able to save the jpeg as a new file.
The send function looks like this
char *parameter_full = new char[f3->picture().size()+2];
sprintf(parameter_full,"%s\n\0",f3->picture().data());
// send
result = send(c,parameter_full,strlen(parameter_full),0);
delete[] parameter_full;
where
f3->picture().data() returns a pointer to the internal data structure (it returns char*) and
f3->picture().size() returns the size of the array.
Then Android receives it with
String imageString = inFromServer.readLine();
byte[] imageBytes = imageString.getBytes();
Bitmap cover = BitmapFactory.decodeByteArray(imageBytes,0,imageBytes.length);
But somehow decodeByteArray always returns null. My idea is that Java doesn't receive the image correctly because imageString only consists of 4 characters...while the extracted jpeg file has a size of 12.7 KB.
But what has gone wrong?
Martin
You shouldn't use string functions on byte data because 0 values are taken as string terminators. Try looking into memcpy on the C++ side if you need to copy the char* and also the byte[] read functions for InputStream on the Java side.