Optimizing Videos with JPEGmini
Our Background
For the past 10 years, JPEGmini has been improving photography workflows by reducing image size without affecting resolution or quality. Ever since the product was introduced to the market, users have been asking us about video support: “Can you apply the same ‘magic’ you use for photos to my video files?” Well, the answer is: Yes! We’ve added video support to the latest version of JPEGmini, so now you can optimize both your photos and your videos!
Video optimization is not new to Beamr, the company that developed JPEGmini. Beamr has released several enterprise products which optimize video files, based on the same technology that powers JPEGmini. These products have been used by operators of video streaming services to reduce their video storage and delivery costs and to improve the streaming experience for their end-users. But now, for the first time, we are adding video optimization capability to JPEGmini, our consumer product line.
The technology behind the image and video optimization solutions developed by Beamr is very unique: The algorithms are covered by 34 granted international patents, and we were recently awarded the 2021 Technology and Engineering Emmy® award from the National Academy of Television Arts & Sciences for this technology.
In this blog post, we’ll explain how the new video optimization feature works under the hood, and what you can expect when using it on your videos. But before we do that, let’s take a look at the image optimization technology first, as this is the basic building block of the technology. After you understand how it works for images, it will be much easier to extend this concept to video optimization.
How Does JPEGmini Work For Photos?
Basically, the way JPEGmini performs its “magic” of reducing file size without affecting quality, is by removing perceptually redundant information from the image file. This is information that is imperceptible by humans, and there is no point in keeping information that we can’t see… But how do we identify such information, and remove it, while making sure all “essential” information is preserved? This is where the Beamr Quality Measure (BQM) comes into play.
BQM is a unique quality metric – it’s a mathematical algorithm, which is highly correlated with the way humans perceive image quality. So BQM can tell us with a very high level of accuracy, whether any visible artifacts exist in our optimized images when compared to the original images. BQM looks at various aspects of the difference between the two images: How the pixel values differ between the input and the optimized image, how much blockiness was added, how has the texture changed, etc. These different attributes are analyzed for each of the image “tiles”, which are small segments of the image, and combined into a single “quality score” for that tile. The tile quality score is then used to control the amount of compression applied to that tile.
More information about BQM can be found in Beamr’s blog post and podcast episode.
JPEGmini relies on BQM in its optimization process. Basically, we take the original JPEG image and apply higher levels of compression to produce several candidate images that are more compressed than the original JPEG. Then we use BQM to compare the quality of the candidate images to the source image. Finally, we determine which is the smallest candidate (in bytes) that does not exhibit any additional visible artifacts when compared to the original JPEG. This is the selected candidate that JPEGmini outputs as the final optimized JPEG. The process is described in the diagram below.
It’s important to note that the “encoding parameters” we use to create the candidate and final JPEG images are not simply an indication of a single “Quality Factor” (QF) number. We actually go much deeper into the JPEG file, and modify a whole set of parameters that is called the “Quantization Matrix”. This set of parameters determine the amount of compression applied to each individual 8×8 pixel block, and we modify this set adaptively, based on the amount of additional compression we wish to apply, and on the characteristics of the source JPEG image. This enables us to create the most compact representation of the image that is possible under the JPEG standard.
As you probably know, JPEGmini is not limited to JPEG images: In JPEGmini 3.0 and beyond, we use a similar technique to create an optimized JPEG from a source HEIC image. In the future, we plan to support additional input and output image formats, including emerging image formats such as AVIF and JPEG XL.
How Does JPEGmini Work For Videos?
Is video just a series of images? Well, yes and no… It turns out that video frames have different characteristics than still images when they undergo video compression, so extending JPEGmini’s capabilities to support video files required some adaptations to BQM. For example, when video frames are compressed, it is possible that sharp edges will be “smeared”, and this type of artifact would be visible to the human eye. In addition, in video it is important to preserve not only the quality of each individual frame, but also the consistency of changes between consecutive frames, which is called “temporal flow”. In order to capture these additional potential artifacts, we enhanced BQM with two more components: An Edge Loss component and a Temporal Consistency component. In addition, when using the video version of BQM, we “pool” together the tile quality scores to create a single quality score for the video frame.
The diagram below shows the structure of the enhanced BQM that we use for video applications.
The method we use to optimize video files is similar to the way we optimize image files, we just apply it to each and every video frame. The video is first decoded, and then each video frame is encoded with higher levels of compression to produce one or more candidate video frames. We then use the video version of BQM to compare the quality of each candidate frame with the source video frame, and determine which is the smallest candidate video frame that, according to BQM, still does not have any visible artifacts compared to the source. The selected candidate frame is then output to the encoded video stream. The diagram below describes the optimization process that we apply to video frames.
By applying the optimization process in a closed-loop, which compares the quality of each output frame with the input, we guarantee that the quality of video is preserved during optimization.
Note that other than changing the compression level of the video frames, JPEGmini does not alter any other attribute of the video file. The file format, metadata, audio track and any other tracks in the files remain untouched and are seamlessly copied and inserted into the output file. This ensures that the video files optimized by JPEGmini retain all of the information in the source video files, and they remain compatible with any video playback or editing software that was used on the source files.
Using The New Video Support in JPEGmini
While JPEG is the image format used for the vast majority of digital images in the past 20 years, for video the situation is more complicated. AVC (also known as H.264) is indeed the most common video codec used across the industry, from capture to download to streaming. However, there are other formats used in specific applications: For example, high-quality video master files used for production and editing are often stored in Apple’s ProRes format. Web video is delivered from YouTube in VP9 format. And the newest smartphones from Apple and Samsung can also capture videos using HEVC, a more efficient codec than AVC.
Since supporting all of these different video formats in the first version of video for JPEGmini would be a huge task, and we wanted to put this feature in the hands of users as fast as possible, we decided to support only AVC, the most common video format, in the first release of JPEGmini with video. AVC is supported in the most common file formats for stored videos, which are MP4 and MOV (Quicktime). So if your phone or camera captured a video file, it will most likely be in MP4/AVC or MOV/AVC formats and supported by JPEGmini. If you see a message that a certain video file is not supported by JPEGmini, it is probably not using AVC as the video format, or MP4 or MOV as the file format.
Other than using an incompatible video codec, a file may also be rejected by JPEGmini if the video track could not be optimized. This happens if the video track, or some parts of it, already contain encoding artifacts. In this case, any further reduction in video bitrate will cause additional visual quality degradation, which is unacceptable for JPEGmini. These video streams, which we refer to as “highly compressed”, cannot be further optimized, so when JPEGmini detects this situation it simply rejects the file. The number of files rejected according to this criteria will vary according to the user’s library content. If the library includes, for example, personal videos that were taken by smartphones that are shaky, or contain a lot of noise since they were taken in a low-light environment, it is not uncommon that 20-30% of the files will be rejected as “highly compressed”. This is actually good news: It means that JPEGmini is committed to quality, and will not optimize a file unless it is sure that the full quality of the video can be preserved.
An important note: If you want to verify that the quality of your video files has been preserved, always do so by playing the original and the optimized video files side-by-side. If you freeze on specific frames, they may look different. But when the videos are played in motion, they will look exactly the same. The quality guarantee of JPEGmini refers to videos played in motion, not to individual video frames that are paused. Also note that resizing (changing the resolution) during optimization, which is an option for Photos, is not supported in the initial release of video for JPEGmini.
So how many % can you expect to shave off your video files? Again, this depends… It can range from nothing (rejected files, as discussed in the previous paragraph), up to 80% and sometimes even more. It depends on the quality of your video files, and the amount of AVC compression that has been applied to them. We found that for high-quality files, we can typically reduce the file size by 30-50%.
You’ve probably noticed that optimizing video files takes much longer than optimizing JPEG photos. The reason is clear: Even a 30 second video file, at 30 frames per second (FPS), contains 900 video frames.Now imagine an hour long video – it has 54,000 video frames! Since we optimize each video frame separately, you can imagine that would take a significant amount of time. But we’ve also applied some very efficient processing techniques to make optimization of video frames much faster than images. A rough rule of thumb is that on a reasonable computer purchased in the last 5 years, optimizing a Full HD file (with a resolution of 1920×1080 pixels) takes about 3 times as long as the file itself. So optimizing a 30 second video takes around a minute and a half, and optimizing a full hour of video would take around 3 hours. If you have a large collection of video files, it’s best to leave JPEGmini running overnight or over the weekend, and make sure your computer is connected to power…
The Big Picture
In this blog post, we took you “behind the scenes” of JPEGmini technology, so you can better understand how JPEGmini does its magic on photos and videos. The key to JPEGmini optimization is the BQM quality measure, and we use it to find the optimal compression level for each photo or video frame. We also explored the various features of video optimization in JPEGmini: What types of files are supported, why certain files are rejected, and what you can expect in terms of savings and processing time.
We hope you are excited as we are with the new video optimization features in JPEGmini, and look forward to your feedback! How did things work for your video files? What’s the biggest benefit you see in reducing the size of your files? We’d love to hear back from you!