x264 Encoding Options
What they are, How to use them
Inside of any video codec, there are all sorts of different features and settings that can be turned on and off and given different values. Using different options can change the picture quality, file size, or encoding time. While the default options x264 applies are rather good, power users can tweak them depending on usage. For example, you might want to use different options for a cartoon than for an epic action-adventure.
How to use them
In the Handbrake GUI, there is an Advanced tab on the main window where you can add x264 options with pretty graphic buttons. From the command line, you can add them with --x264opts or -x. They are set up the same way as x264 options in MEncoder. That means they follow this format:
option1=value1:option2=value2:option3=value3
That is to say, each option's value is assigned with an equals sign (=), and each option-value pair is separated by a colon (:). One simple example would be:
bframes=6:me=umh:subme=5
Don't worry about what "bframes" or "me" or "umh" mean, quite yet. A full description of the options is provided further down in this guide.
Options explained
Frames
Reference Frames (ref)
Sane values are 1-6. The more you add, the higher the quality — but the slower the encode. Be careful...too many and QuickTime struggle to play the video back.
Mixed References (mixed-refs)
With this on, different references can be used for different parts of each macroblock, increasing quality.
B-Frames (bframes)
Sane values are 1-6. B-Frames are smaller than other frames, so they let you pack in more quality at the same bitrate. Use more of them with animated material: 9-16.
Direct Prediction (direct)
Direct prediction tells x264 what method to use when guessing motion for certain parts of a B-frame. It can either look at other parts of the current frame (spatial) or compare against the following P-frameframe (temporal). You're best off setting this to automatic, so x264 decides which method is best on its own. Don't select none assuming it will be faster; instead it will take longer and look worse. If you're going to choose between spatial and temporal, spatial is usually better.
Weighted B-Frames (weightb)
Sometimes x264 will base a B-frame's motion compensation on frames both before and after. With weighted B-frames, the amount of influence each frame has is related to its distance from the frame being encoded, instead of both having equal influence.
Pyramidal B-Frames (b-pyramid)
B-frame pyramids are a High Profile feature. This means that if you enable it, YOUR VIDEO WILL NOT PLAY IN QUICKTIME. Pyramidal B-frames mean that B-frames don't just reference surrounding reference frames — instead, it also treats a previous B-frame as a reference, improving quality/lowering bitrate at the expense of complexity. Logically, to reference an earlier B-frame, you must tell x264 to use at least 2 B-frames.
Motion
Motion Estimation Method (me)
Controls the motion estimation method. At the most basic setting, dia, x264 will only consider a diamond-shaped region around each pixel. The default setting, hex, is similar to dia but uses a hexagon shape. Uneven multi-hexagon, umh, searches a number of different patterns across a wider area and thus is slower than hex and dia but further increases compression efficiency and quality. esa, an exhaustive search of a square around each pixel (whose size is controlled by the me-range parameter), is only intended for experimentation and testing of algorithms and should not be used, as umh will generally produce as good or better results in vastly less time.
Motion Estimation Range (me-range)
This range is the radius, in pixels, x264 should use for motion estimation searches. It only has an effect when you use Uneven Multi-Hexagonal or Exhaustive searching. 24, 32, and 64 are good values.
Subpixel Motion Estimation (subme)
This setting is finer-grained than the motion estimation settings above. Instead of dealing with whole pixels, it deals with 4 fractional pixels, or quarter pixels (qpel). Higher levels increase quality by further refining the motion prediction for these quarter pixels, but take longer to encode. The default, 6, turns on a feature called rate distortion optimization, including psychovisual enhancements. 7 enables that rate distortion for B-frames. 8 refines those decisions for I and P frames, and 9 adds on refinement for B-frames as well.
Analysis
Analysis (analyse)
Analysis controls how finely x264 divides up a frame to capture detail. Full macroblocks are 16x16 pixels, but x264 can go down all the way to 4x4 blocks if it judges it necessary. By default it only breaks up key frames that much. To give x264 the freedom to make the best decisions for all frames, use "all" analysis. If you want to create a high profile H.264 video (which is less compatible with the world at large than main profile), also check the "8x8 DCT blocks" box to add yet another block size for analysis.
8x8 DCT Blocks (8x8dct)
When Analysis is set to "all," checking this box lets x264 break key frames down into 8x8 blocks of pixels for analysis. This is a high profile feature of H.264, which makes it less compatible. It should slightly decrease bitrate or improve quality.
Other
Deblocking (filter)
x264 includes an in-loop deblocking filter. What this means is that blocky compression artifacts are smoothed away when you play back the video. It has two settings: strength and threshold, just like a simple filter in Photoshop. Strength controls the amount of deblocking applied to the whole frame. If you drop down below 0, you reduce the amount of blurring. Go too negative, and you'll get an effect somewhat like oversharpening an image. Go into positive values, and the image may become too soft. Threshold controls how sensitive the filter is to whether something in a block is detail that needs to be preserved: lower numbers blur details less. The default deblocking values are 0 and 0. This does not mean zero deblocking. It means x264 will apply the regular deblocking strength and thresholds the codec authors have selected as working the best in most cases. While many, many people stick with the default deblocking values of 0,0, other people disagree. Some prefer a slightly less blurred image for live action material, and use values like -2,-1 or -2,-2. Others will raise it to 1,1 or even 3,3 for animation. While the values for each setting extend from -6 to 6, the consensus is that going below -3 or above 3 is worthless.
Trellis (trellis)
Trellis fine-tunes how bitrate is doled out, so it can reduce file size/bitrate or increase quality. A value of 2 forces it to be used more often than a value of 1, and should only be used for a 2nd pass.
No DCT Decimate (no-dct-decimate)
To save space, x264 will "zero out" blocks when it thinks they won't be perceptible by the viewer. This negligibly reduces quality, but in rare cases it can mess up and produce visible artifacts. This situation can be alleviated by telling x264 not to decimate DCT blocks. It increases quality but also bitrate/file size, so if you use it when you've specified a target bitrate you will end up with a worse picture than without it. However, when used with constant quality encoding, or if you boost the average bitrate to compensate, you might get a better result.
No Fast P-Skip (no-fast-pskip)
This can help with blocking on solid colors like blue skies, but it also slows down the encode.
CABAC Entropy (cabac)
CABAC, or context adaptive binary arithmetic coding, is used by x264 to reduce the bitrate needed for a given quality by 15%. This makes it very cool and very useful, and it should be left on whenever possible. However, it is incompatible with the iPod 5.5G, and makes the AppleTV struggle. So turn it off for those. CABAC is a kind of entropy coding, which means that it compresses data by making shorthand symbols to represent long streams of data. The "entropy" part means that the symbols it uses the most often are the smallest. When you disable CABAC, another entropy coding scheme gets enabled, called CAVLC (context adaptive variable-length coding). CAVLC is a lot less efficient, which is why it needs 15% more bitrate to achieve the same quality as CABAC.
Custom Quantization Matrix (cqm)
QuickTime incompatible -- A custom quantization matrix sounds intimidating, but all you really need to know is that it can improve picture quality. Some video gurus swear by them. Basically, the matrices allow variation in the amount of compression different types of visual information (noise, edges, textures...) receive. Using flat is the same as not applying a CQM. Just save them as text files, and enter the full path to the matrix as the option value.
Examples
High Profile
High Profile enables all the bells and whistles in the H.264 standard. It is then that the codec truly shines and proves it is superior to plain MPEG-4.
Here are my High Profile settings. Note that it does not take advantage of every high profile feature, like a custom quantization matrix. Use it with 2-pass at a medium-to-high average bitrate (1500, plus or minus 500kbps):
ref=3:mixed-refs=1:bframes=3:b-pyramid=1:b-rdo=1:bime=1:weightb=1:subme=6:trellis=1:analyse=all:8x8dct=1:vbv-maxrate=25000
However, you might want something a bit more excessive, like this:
ref=6:mixed-refs=1:bframes=3:b-pyramid=1:weightb=1:subme=7:trellis=2:analyse=all:8x8dct=1:no-fast-pskip=1:no-dct-decimate=1:me=umh:merange=64:filter=-2,-1:direct=auto
Description: A caustic simulation in Cinema 4D
Interface designer, Art Director at AppDynamics (16'—20'). Creative Director in BioTech (20' — Present).
