I'm trying to make an Azure Function in Java. I need to make an excel file and upload it in BLOB container.
When I build the project and the tests start, then it works without problems and it uploads the file in the container, when instead I debug the project or I deploy it on Azure and I run it via internet (calling the service), it doesn't upload it. It blocks when it tries to upload the file.
Can you help me please? I'm on this problem since a few days.
Thank you.
I attach the method where it uploads the file:
#FunctionName("FunctionTest")
public HttpResponseMessage run(
#HttpTrigger(
name = "req",
methods = {HttpMethod.GET, HttpMethod.POST},
authLevel = AuthorizationLevel.ANONYMOUS)
HttpRequestMessage<Optional<String>> request,
final ExecutionContext context) {
context.getLogger().info("Java HTTP trigger processed a request.");
final String queryAccountName = request.getQueryParameters().get("AccountName");
String accountName = request.getBody().orElse(queryAccountName);
final String queryAccountKey = request.getQueryParameters().get("AccountKey");
String accountKey = request.getBody().orElse(queryAccountKey);
context.getLogger().info("Azure Blob storage v12 - Java quickstart sample\n");
// Retrieve the connection string for use with the application. The storage
// connection string is stored in an environment variable on the machine
// running the application called AZURE_STORAGE_CONNECTION_STRING. If the environment variable
// is created after the application is launched in a console or with
// Visual Studio, the shell or application needs to be closed and reloaded
// to take the environment variable into account.
// String connectStr = System.getenv("AZURE_STORAGE_CONNECTION_STRING");
//String connectStr = "DefaultEndpointsProtocol=https;AccountName="+accountName+";AccountKey="+accountKey+";EndpointSuffix=core.windows.net";
// Create a BlobServiceClient object which will be used to create a container client
//BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().connectionString(connectStr).buildClient();
StorageSharedKeyCredential credential = new StorageSharedKeyCredential(accountName, accountKey);
String endpoint = String.format(Locale.ROOT, "https://%s.blob.core.windows.net", accountName);
BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().endpoint(endpoint).credential(credential).buildClient();
//Create a unique name for the container
String containerName = "container-name";
// Create the container and return a container client object
//BlobContainerClient containerClient = blobServiceClient.createBlobContainer(containerName);
BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(containerName);
// Create a local file in the ./data/ directory for uploading and downloading
/*String pathFile = "./data/";
String fileName = "quickstart" + java.util.UUID.randomUUID() + ".txt";
File localFile = new File(pathFile + fileName);
// Write text to the file
FileWriter writer;
try {
writer = new FileWriter(pathFile + fileName, true);
writer.write("Hello, World!");
writer.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}*/
// Get a reference to a blob
// Upload the blob
String pathFile = System.getenv("TEMP") + "\\";
String fileName = creaReport(context)+".xlsx"; // creating file Excel - IT DOESN'T EVEN WORK WITH TXT FILE
BlobClient blobClient = containerClient.getBlobClient(fileName);
System.out.println("\nUploading to Blob storage as blob:\n\t" + blobClient.getBlobUrl());
blobClient.uploadFromFile(pathFile + fileName, true); // IT BLOCKS HERE
System.out.println("\nListing blobs...");
// List the blob(s) in the container.
for (BlobItem blobItem : containerClient.listBlobs()) {
System.out.println("\t" + blobItem.getName());
}
// Download the blob to a local file
// Append the string "DOWNLOAD" before the .txt extension so that you can see both files.
//String downloadFileName = fileName.replace(".txt", "DOWNLOAD.txt");
String downloadFileName = fileName.replace(".xlsx", "DOWNLOAD.xlsx");
File downloadedFile = new File(pathFile + downloadFileName);
System.out.println("\nDownloading blob to\n\t " + pathFile + downloadFileName);
blobClient.downloadToFile(pathFile + downloadFileName, true);
// Clean up
System.out.println("\nPress the Enter key to begin clean up");
System.console().readLine();
/*System.out.println("Deleting blob container...");
containerClient.delete();*/
System.out.println("Deleting the local source and downloaded files...");
localFile.delete();
downloadedFile.delete();
System.out.println("Done");
return request.createResponseBuilder(HttpStatus.OK).body("Blob uploaded").build();
}
For this problem, I test it in my side and summarize the point as below:
The reason for this problem is the files in local/temp are not shared among site instances. You can refer to this page.
So I met the same problem as you, I deploy my java function to azure and add a file under local/temp path manually. Then run the function, it can't access the data of the file.
After that, I edit my function code. Create a txt file by the code below in my function:
String filePath="d:\\local\\Temp\\test1.txt";
File file = new File(filePath);
try {
FileWriter writer = new FileWriter(file);
writer.write("Test data");
writer.close();
} catch (IOException e1) {
context.getLogger().info(e1.getMessage());
e1.printStackTrace();
}
And then read the file in the same function by the code below:
InputStream is = null;
int i;
char c;
try {
is = new FileInputStream(filePath);
while ((i = is.read()) != -1) {
c = (char) i;
context.getLogger().info("===inputstream==" + c);
}
} catch (Exception e) {
context.getLogger().info("===try catch error");
e.printStackTrace();
} finally {
if (is != null) {
try {
is.close();
} catch (IOException e) {
context.getLogger().info("===finally error");
e.printStackTrace();
}
}
}
Deploy the code to azure, the function app run in a consumption plan(so it will just use one instance if I just run it once). Running it, I read the data of the file success.
According to the test above, I suggest you do not create the file in local/temp. You'd better create the file in d:\home\site\wwwroot, you can create a folder under wwwroot and create the files in the folder. I test it works fine.
Hope it helps~
Related
I have a simple Spring Boot microservice that takes care of uploading, retrieving and deleting images to/from Google Cloud Storage. I have the following code for the get request in my service:
public StorageObject getImage(String fileName) throws IOException {
StorageObject object = storage.objects().get(bucketName, fileName).execute();
File file = new File("./" + fileName);
FileOutputStream os = new FileOutputStream(file);
storage.getRequestFactory()
.buildGetRequest(new GenericUrl(object.getMediaLink()))
.execute()
.download(os);
object.set("file", file);
return object;
}
And this is my controller part:
#GetMapping("/get/image/{id}")
public ResponseEntity<byte[]> getImage(#PathVariable("id") Long id) {
try {
String fileName = imageService.findImageById(id);
StorageObject object = gcsService.getImage(fileName);
byte[] res = Files.toByteArray((File) object.get("file"));
return ResponseEntity.ok()
.contentType(MediaType.IMAGE_JPEG)
.body(res);
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException("No such file or directory");
}
}
It all works fine in terms of getting the image in the response, but my problem is that the images get downloaded at the root directory of the project too. Many images are going to be uploaded through this service so this is an issue. I only want to display the images in the response (as a byteArray), without having them download. I tried playing with the code but couldn't manage to get it to work as I want.
I'd suggest to instead stream the download, while skipping the FileChannel operation:
public static void streamObjectDownload(
String projectId, String bucketName, String objectName, String targetFile
) {
Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
try (ReadChannel reader = storage.reader(BlobId.of(bucketName, objectName));
FileChannel targetFileChannel = FileChannel.open(Paths.get(targetFile), StandardOpenOption.WRITE)) {
ByteStreams.copy(reader, targetFileChannel);
System.out.println(
"Downloaded object " + objectName
+ " from bucket " + bucketName
+ " to " + targetFile
+ " using a ReadChannel.");
}
} catch (IOException e) {
e.printStacktrace()
}
}
One can eg. obtain a FileChannel from a RandomAccessFile:
RandomAccessFile file = new RandomAccessFile(Paths.get(targetFile), StandardOpenOption.WRITE);
FileChannel channel = file.getChannel();
While the Spring framework similarly has a GoogleStorageResource:
public OutputStream getOutputStream() throws IOExceptionReturns the output stream for a Google Cloud Storage file.
Then convert from OutputStream to byte[] (this may be binary or ASCII data):
byte[] bytes = os.toByteArray();
Would it work for you to create Signed URLs in Cloud Storage to display your images? These URLs give access to storage bucket files for a limited time, and then expire, so you would rather not store temporary copies of the image locally as is suggested in this post.
How do I insert a text file into an existing zip file where I have a stream of both the files.
Sample code is below:
I am using IOutils.copy(in, out), but it is replacing the content. I need to insert the file and keep the existing content.
public URI createACopyOfFile(String requestId) throws Exception {
BlobOutputStream blobOutputStream = null;
BlobInputStream blobInputStream = null;
try {
long startTime = System.currentTimeMillis();
System.out.print("Start createACopyOfFile ->" + startTime);
long endTime = 0;
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient cloudBlobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = cloudBlobClient.getContainerReference("tme-container");
container.createIfNotExists();
String blobName = "dummy.zip";
CloudBlockBlob sourceBlob = container.getBlockBlobReference(blobName);
CloudBlockBlob targetBlob = container.getBlockBlobReference(requestId + "/dummy_copy.zip");
targetBlob.startCopyFromBlob(sourceBlob);
blobOutputStream = targetBlob.openOutputStream();
CloudBlockBlob blob = container.getBlockBlobReference(requestId + "/" + "activation.txt");
blob.uploadText("Hello world");
blobInputStream = blob.openInputStream();
IOUtils.copy(blobInputStream, blobOutputStream);
endTime = System.currentTimeMillis();
System.out.print("End createACopyOfFile -> " + endTime);
System.out.println("Total time taken for createACopyOfFile - > " + (endTime - startTime) / 1000);
return targetBlob.getUri();
} finally {
try {
if (blobInputStream != null)
blobInputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
if (blobOutputStream != null)
blobOutputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I want to tell you that there is no ways to do it by azure SDK. When you add file to the zip file, the file will
reorganize its content and generate a new zip file.
There is one method that you can download the .zip file and append file in local by JAVA. After then, you can upload your file to azure storage again. But your file is a big file, so it's not a feasible method.
The references:
Appending files to a zip file with Java
Hope it helps. Any concerns, please feel free to let me know.
I have a java program that call my Perl script to upload a file. It has a file parameter to the Perl script that contain the location of file to upload.
public static void legacyPerlInspectionUpload(String creator, String artifactId, java.io.File uploadedFile, String description ) {
PostMethod mPost = new PostMethod(getProperty(Constants.PERL_FILE_URL) + "inspectionUpload.pl");
try {
String upsSessionId = getUpsSessionCookie();
//When passing multiple cookies as a String, seperate each cookie with a semi-colon and space
String cookies = "UPS_SESSION=" + upsSessionId;
log.debug(getCurrentUser() + " Inspection File Upload Cookies " + cookies);
Part[] parts = {
new StringPart("creator", creator),
new StringPart("artifactId", artifactId),
new StringPart("fileName", uploadedFile.getName()),
new StringPart("description", description),
new FilePart("fileContent", uploadedFile) };
mPost.setRequestEntity(new MultipartRequestEntity(parts, mPost.getParams()));
mPost.setRequestHeader("Cookie",cookies);
HttpClient httpClient = new HttpClient();
int status = httpClient.executeMethod(mPost);
if (status == HttpStatus.SC_OK) {
String tmpRetVal = mPost.getResponseBodyAsString();
log.info(getCurrentUser() + ":Inspection Upload complete, response=" + tmpRetVal);
} else {
log.info(getCurrentUser() + ":Inspection Upload failed, response=" + HttpStatus.getStatusText(status));
}
} catch (Exception ex) {
log.error(getCurrentUser() + ": Error in Inspection upload reason:" + ex.getMessage());
ex.printStackTrace();
} finally {
mPost.releaseConnection();
}
}
In this part of my Perl script, it get the information about the file, read from it and write the content to a blink file in my server.
#
# Time to upload the file onto the server in an appropropriate path.
#
$fileHandle=$obj->param('fileContent');
writeLog("fileHandle:$fileHandle");
open(OUTFILE,">$AttachFile");
while ($bytesread=read($fileHandle,$buffer,1024)) {
print OUTFILE $buffer;
}
close(OUTFILE);
writeLog("Download file, checking stats.");
#
# Find out if the file was correctly uploaded. If it was not the file size will be 0.
#
($size) = (stat($AttachFile))[7];
Right now the problem is this only work for file with no space in its name, otherwise $size is 0. I was reading online and it seems both Java file and Perl filehandle work with space, so what am I doing wrong?
Your poor variable naming has tripped you up:
open(OUTFILE,">$AttachFile");
^^^^^^^---this is your filehandle
while ($bytesread=read($fileHandle,$buffer,1024)) {
^^^^^^^^^^^--- this is just a string
You're trying to read from something that's NOT a filehandle, it's just a variable whose name happens to be "filehandle". You never opened up the specified file for reading. e.g. you're missing
open(INFILE, "<$fileHandle");
read(INFILE, $buffer, 1024);
I'm using Apache Commons FTP to upload a file. Before uploading I want to check if the file already exists on the server and make a backup from it to a backup directory on the same server.
Does anyone know how to copy a file from a FTP server to a backup directory on the same server?
public static void uploadWithCommonsFTP(File fileToBeUpload){
FTPClient f = new FTPClient();
FTPFile backupDirectory;
try {
f.connect(server.getServer());
f.login(server.getUsername(), server.getPassword());
FTPFile[] directories = f.listDirectories();
FTPFile[] files = f.listFiles();
for(FTPFile file:directories){
if (!file.getName().equalsIgnoreCase("backup")) {
backupDirectory=file;
} else {
f.makeDirectory("backup");
}
}
for(FTPFile file: files){
if(file.getName().equals(fileToBeUpload.getName())){
//copy file to backupDirectory
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
Edited code: still there is a problem, when i backup zip file, the backup-ed file is corrupted.
Does any body know the reason for it?
public static void backupUploadWithCommonsFTP(File fileToBeUpload) {
FTPClient f = new FTPClient();
boolean backupDirectoryExist = false;
boolean fileToBeUploadExist = false;
FTPFile backupDirectory = null;
try {
f.connect(server.getServer());
f.login(server.getUsername(), server.getPassword());
FTPFile[] directories = f.listDirectories();
// Check for existence of backup directory
for (FTPFile file : directories) {
String filename = file.getName();
if (file.isDirectory() && filename.equalsIgnoreCase("backup")) {
backupDirectory = file;
backupDirectoryExist = true;
break;
}
}
if (!backupDirectoryExist) {
f.makeDirectory("backup");
}
// Check if file already exist on the server
f.changeWorkingDirectory("files");
FTPFile[] files = f.listFiles();
f.changeWorkingDirectory("backup");
String filePathToBeBackup="/home/user/backup/";
String prefix;
String suffix;
String fileNameToBeBackup;
FTPFile fileReadyForBackup = null;
f.setFileType(FTP.BINARY_FILE_TYPE);
f.setFileTransferMode(FTP.BINARY_FILE_TYPE);
for (FTPFile file : files) {
if (file.isFile() && file.getName().equals(fileToBeUpload.getName())) {
prefix = FilenameUtils.getBaseName(file.getName());
suffix = ".".concat(FilenameUtils.getExtension(file.getName()));
fileNameToBeBackup = prefix.concat(Calendar.getInstance().getTime().toString().concat(suffix));
filePathToBeBackup = filePathToBeBackup.concat(fileNameToBeBackup);
fileReadyForBackup = file;
fileToBeUploadExist = true;
break;
}
}
// If file already exist on the server create a backup from it otherwise just upload the file.
if(fileToBeUploadExist){
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
f.retrieveFile(fileReadyForBackup.getName(), outputStream);
InputStream is = new ByteArrayInputStream(outputStream.toByteArray());
if(f.storeUniqueFile(filePathToBeBackup, is)){
JOptionPane.showMessageDialog(null, "Backup succeeded.");
f.changeWorkingDirectory("files");
boolean reply = f.storeFile(fileToBeUpload.getName(), new FileInputStream(fileToBeUpload));
if(reply){
JOptionPane.showMessageDialog(null,"Upload succeeded.");
}else{
JOptionPane.showMessageDialog(null,"Upload failed after backup.");
}
}else{
JOptionPane.showMessageDialog(null,"Backup failed.");
}
}else{
f.changeWorkingDirectory("files");
f.setFileType(FTP.BINARY_FILE_TYPE);
f.enterLocalPassiveMode();
InputStream inputStream = new FileInputStream(fileToBeUpload);
ByteArrayInputStream in = new ByteArrayInputStream(FileUtils.readFileToByteArray(fileToBeUpload));
boolean reply = f.storeFile(fileToBeUpload.getName(), in);
System.out.println("Reply code for storing file to server: " + reply);
if(!f.completePendingCommand()) {
f.logout();
f.disconnect();
System.err.println("File transfer failed.");
System.exit(1);
}
if(reply){
JOptionPane.showMessageDialog(null,"File uploaded successfully without making backup." +
"\nReason: There wasn't any previous version of this file.");
}else{
JOptionPane.showMessageDialog(null,"Upload failed.");
}
}
//Logout and disconnect from server
in.close();
f.logout();
f.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}
If you are using apache commons net FTPClient, there is a direct method to move a file from one location to another location (if the user has proper permissions).
ftpClient.rename(from, to);
or, If you are familiar with ftp commands, you can use something like
ftpClient.sendCommand(FTPCommand.yourCommand, args);
if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode())) {
//command successful;
} else {
//check for reply code, and take appropriate action.
}
If you are using any other client, go through the documentation, There wont be much changes between client implementations.
UPDATE:
Above approach moves the file to to directory, i.e, the file won't be there in from directory anymore. Basically ftp protocol meant to be transfer the files from local <-> remote or remote <-> other remote but not to transfer with in the server.
The work around here, would be simpler, get the complete file to a local InputStream and write it back to the server as a new file in the back up directory.
to get the complete file,
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ftpClient.retrieveFile(fileName, outputStream);
InputStream is = new ByteArrayInputStream(outputStream.toByteArray());
now, store this stream to backup directory. First we need to change working directory to backup directory.
// assuming backup directory is with in current working directory
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);//binary files
ftpClient.changeWorkingDirectory("backup");
//this overwrites the existing file
ftpClient.storeFile(fileName, is);
//if you don't want to overwrite it use storeUniqueFile
Hope this helps you..
Try this way,
I am using apache's library .
ftpClient.rename(from, to) will make it easier, i have mentioned in the code below
where to add ftpClient.rename(from,to).
public void goforIt(){
FTPClient con = null;
try
{
con = new FTPClient();
con.connect("www.ujudgeit.net");
if (con.login("ujud3", "Stevejobs27!!!!"))
{
con.enterLocalPassiveMode(); // important!
con.setFileType(FTP.BINARY_FILE_TYPE);
String data = "/sdcard/prerakm4a.m4a";
ByteArrayInputStream(data.getBytes());
FileInputStream in = new FileInputStream(new File(data));
boolean result = con.storeFile("/Ads/prerakm4a.m4a", in);
in.close();
if (result)
{
Log.v("upload result", "succeeded");
//$$$$$$$$$$$$$$$$$$$$$$$$$$$$$Add the backup Here$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$//
// Now here you can store the file into a backup location
// Use ftpClient.rename(from, to) to place it in backup
//$$$$$$$$$$$$$$$$$$$$$$$$$$$$$Add the backup Here$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$//
}
con.logout();
con.disconnect();
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
There's no standard way to duplicate a remote file over FTP protocol. Some FTP servers support proprietary or non-standard extensions for this though.
So if your are lucky that your server is ProFTPD with mod_copy module, you can use FTP.sendCommand to issue these two commands:
f.sendCommand("CPFR sourcepath");
f.sendCommand("CPTO targetpath");
The second possibility is that your server allows you to execute arbitrary shell commands. This is even less common. If your server supports this you can use SITE EXEC command:
SITE EXEC cp -p sourcepath targetpath
Another workaround is to open a second connection to the FTP server and make the server upload the file to itself by piping a passive mode data connection to an active mode data connection. Implementation of this solution (in PHP though) is shown in FTP copy a file to another place in same FTP.
If neither of this works, all you can do is to download the file to a local temporary location and re-upload it back to the target location. This is that the answer by #RP- shows.
See also FTP copy a file to another place in same FTP.
To backup at same Server (move), can you use:
String source="/home/user/some";
String goal ="/home/user/someOther";
FTPFile[] filesFTP = cliente.listFiles(source);
clientFTP.changeWorkingDirectory(goal); // IMPORTANT change to final directory
for (FTPFile f : archivosFTP)
{
if(f.isFile())
{
cliente.rename(source+"/"+f.getName(), f.getName());
}
}
I am hosting a website on Tomcat server. The application uses Struts 1.1 and Spring for all its operations. I have a page that is used for uploading files to the server.
When user uploads any file it is successfully uploaded but gives a 404 error when tried to retrieve. I checked the file using SSH login, the uploaded file is present in that location. I am scratching my head over this problem from past 4 days but no solution. Its works properly without any problems in my local machine. The problem in there in the deployment.
An Important note: From SSH login, If i try to move that file to some other location and then place it back to its original location, i am able to retrieve the file..!!! I don't know why but I can't do this for every file uploaded by the user. So i modified the my code so that the file is uploaded to a temp location first and then moving it to the correct location. But even this is not working.
FileOutputStream outputStream = null;
FormFile formFile = null;
String tempFilePath = getServlet().getServletContext()
.getRealPath("/")
+ "uploads"
+ System.getProperty("file.separator") + "temp";
try
{
formFile = uploadForm.getFile();
boolean errorflag = false;
if(formFile.getFileSize() > 10660000)
{
request.setAttribute("error",
"File size cannot exceed 10MB!");
errorflag = true;
}
else
{
errorflag = validateFileUpload(request,
formFile, errorflag);
}
if(errorflag)
{
return gotoKnowledgeSharingPage(mapping,
request, actionHelper, session, userid,
instid);
}
File folder = new File(tempFilePath);
if(!folder.exists())
{
folder.mkdir();
}
outputStream = new FileOutputStream(new File(
tempFilePath, formFile.getFileName()));
outputStream.write(formFile.getFileData());
}
finally
{
if(outputStream != null)
{
outputStream.flush();
outputStream.close();
}
}
String finalFilePath = getServlet().getServletContext()
.getRealPath("/")
+ "uploads"
+ System.getProperty("file.separator")
+ session.getAttribute("userid");
//+ System.getProperty("file.separator")
// + formFile.getFileName();
File oldPath = new File(tempFilePath
+ System.getProperty("file.separator")
+ formFile.getFileName());
// Move file to new directory
File newPath = new File(finalFilePath);
if(!newPath.exists())
{
newPath.mkdir();
}
boolean success = oldPath.renameTo(new File(
finalFilePath, formFile.getFileName()));
if(success)
{
actionHelper.insertIntoUploadTable(userid,
knowledgeForm, formFile.getFileName());
}
else
{
if(oldPath.exists())
{
oldPath.delete();
}
}