I'm trying to understand process of parsing h.264 NAL units (to extract information about slices, macroblocks etc.), so I'm writing simple bit stream parser for h.264
Are there any example (training) files, which, for example, contains single NAL unit or single slice?
Does anybody knows where I can get such training data?
Thanks
If you want training data, you can download the H.264 reference software from http://iphome.hhi.de/suehring/tml/download/. Note that this reference software is written in c++ though. You don't need to be well versed in c++ though, you do need to be able to build the encoder and then you can use it as a tool to generate .264 data.
The bin directory contains .yuv files (raw uncompressed) and using the configuration files you can then generate .264 files. If you want a single NAL Unit as you specified, you can configure the encoder via the configuration file to only encode a single video frame using the FramesToBeEncoded parameter. If you open the generated .264 using a hex editor you can identify the NAL units by their start codes. By adapting the configuration files, you should be able to generate your desired test data.
Note that even if you only generate one frame, there might be more than one NAL unit inside the .264 file since the sequence and picture parameter sets are prepended to the IDR frame. You could easily isolate and separate e.g in c++ by searching for the start codes.
Related
I have an InputStream of data that is the content of a file, but does not have any file information attached. I would like to be able to distinguish between cases when the data represents a *.zip file, and cases where it is a container file format (e.g. *.docx, *.odt, *.jar) that uses zip under the covers. I don't necessarily need to know what the container format is, just whether a stream is a "plain" zip or not (so I know whether it's appropriate to split the stream into separate files or not).
Is this possible? I'm happy to do the detection either after decompressing or before.
Ideally I'm trying to do this in Java, but if there are code examples in other languages then I'm happy to port them across if necessary.
There's no absolutely reliable and correct way to do this, because those formats that use the ZIP format as a container tend to be 100% valid and correct ZIP files.
So they are ZIP files.
However, since there's not an infinite number of those formats (and only a smaller subset of those are commonly found in the real world), you can probably get away with just specifically detecting those formats and treating everything that you don't recognize as a "real" ZIP file.
Most of these formats require some kind of easy-to-check identifier in the early bytes of the file, so if you are okay with writing specification-specific code it should be easy enough.
file detects most of those formats correctly, so looking into its source should give you enough pointers.
Some examples:
OpenDocument files (this file contains all kinds of archives, not just ODx files).
Office Open XML files
It's also quite likely (haven't checked) that Apache Tika already does all that detection.
I am learning to build neural nets and I came across this code on github,
https://github.com/PavelJunek/back-propagation-java
There is a training set and a validation set required to be used but I don't know where to input the files. The Readme doesn't quite explain how to use the files. How do I test with different csv files I have on this code?
How so? It tells you exactly what to do. The program needs to get two CSV files: a CSV file containing all the training data and a second CSV file containing all of the validation data.
If you have a look at the Program.java file (in the main method), you'll see that you need to pass both files as arguments with the command line.
In migrating from a CMS that stored files in the database, over to a system that stores them in AWS S3, I cant seem to find any options other than reverse engineering the format from Java (the old system) and implementing it all myself from scratch in python, using either the java code or rfc1867 as a reference.
I have database dumps containing long strings of encoded files.
I'm not 100% clear which binary file upload encoding has been used. But there is consistency between the first characters of each file types.
UEsDBBQA is the first 8 characters in a large number of the DOCX file formats, and UEsDBBQABgAIAAAA is the first 16 characters in more than 75% of the DOCX files.
JVBERi0xLj is the first 10 characters of many of the PDF files.
Every web application framework that allows file uploads has to decode these... so its a known problem. But I cannot find a way to decode these strings with either Python (my language of choice), or with some kind of command line decoding tool...
file doesnt recognise them.
hachoir doesnt recognise them.
Are there any simple tools I can just install, I dont care if they are in C, Perl, Python, Ruby, JavaScript or Mabolge, I just want a tool that can take the encoded string as input (file, stdin, I don't care) and output the decoded original files.
Or am I overthinking the algorithm to decode these files and it would be simpler than it looks and someone can show me how to decode them using pure python?
Most commonly used encoding algorithm to represent binary data as text is Base64. I just did a quick test on a PDF file in Java and I got exactly the same header character sequence when Base64-encoding it.
byte[] bytes = Files.readAllBytes(Paths.get("/test/test.pdf"));
String base64 = DatatypeConverter.printBase64Binary(bytes);
System.out.println(base64.substring(0, 10)); // JVBERi0xLj
So, you're most likely looking for a Base64 decoder.
I don't do Python, so here's a Google search suggestion and the first Stack Overflow link which appeared in the search results to date: Python base64 data decode.
I have REST service that allows users to upload images and then serves those images. The images are stored to a database. I'd like to do JPG optimization for these images.
There's several command line tools to do this but I'd like to do it without first saving them to disk and then running some command-line tools. I'd rather use some Java library to directly operate on a binary stream that contains the image data.
What I'm after is a treatment similar to what for example Trimage does:
Remove all EXIF metadata from the image
Losslessy (re)compressed to the highest available compression levels
Is this possible?
It is possible to remove extraneous APPn and COM markers from the data stream. You can do that without expanding and recompressing.
Each time you expand and decompress with different quantization tables, you loose data in JPEG. There is no real point in decompression.
Yes to question #1. No to question #2.
Is it possible to merge equally sized mp3 format files, and then retrieve, modify each unit file and add new ones in Java? Is there any tool or programming solution?
MP3 format does not allow clean merging of two files to create a new one without re-encoding. The reason is that first and last frame of the file contain some junk information that has to be discarded. You still can merge the files, like here, but it will not be gapless and accurate. Strip id3v1 tag from the first file (last 120 bytes, if it exists), id3v2 from the second file (see this link to see how to find and get size of id3v2), and then merge the files. Things could get complicated if there are LAME frames. But most player should be able to handle these files.
There is nothing built in to Android to achieve that, because there is no MP3 encoder in the SDK. You'll need an MP3 codec library.
See this answer.
Hope the below link helps you to get what you are looking for
Audacity
http://audacity.sourceforge.net/