The FFmpeg library is used by several of the OPAL video plugin codecs. The history of FFmpeg is long and torturous, and there are many API breaks along the way. That said it is the best game in town for performance and interoperability.
The x264 library is only the H.264 encoder, the decoder is in FFmpeg. Note that this is GPL and is not compatible with the MPL license of PTLib/OPAL. As a work around a GPL application "x264plugin_helper" was created and OPAL talks to it via a socket. While in breach of the spirit of GPL, it is legal. And don't get us started on licensing issues ...
Building and installing FFmpeg/x264 on Linux
FFmpeg and x264 are usually available as packages in the distribution. Installing the package and running OPAL configure is usually sufficient. It will be detected and compiled as required. The exact name of the package varies depending on the distribution. For example, on Red Hat it is "sudo yum install ffmpeg-devel x264-devel".
Building and installing FFmpeg/x264 on Windows
Most importantly, don't try to compile FFMpeg or x264 for Windows unless you enjoy pain.
Seriously. Many have tried and failed.
Using a pre-compiled version of FFmpeg
For hard-core fanatics only.
FFmpeg can only be built on Windows using the Msys+MingW toolchain. The FFMpeg on Windows Wiki gives detailed instructions on how to do this.
IMPORTANT: You will need to add "c:\msys" to the VSNET2005_PWLIB_CONFIGURE_EXCLUDE_DIRS environment variable to avoid Opal configure from attempting to use the new header files.
Note 1: If you experience a "hang" while running the "./configure" for FFmpeg, try opening the Task Manager and killing the "lvprcsrv.exe" process. This is apparently a known issue when compiling with the MingW toolchain.
Note 2: If the compile fails with errors relating to "vfw", add the "--disable-devices" option when confguring FFmpeg