Getting source code

First of all, you need to get source code of FFmpeg distribution. You can do this with git. Go to http://ffmpeg.org download site and choose clone URL. Then clone repository using git:

$ git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg
Cloning into 'ffmpeg'...
remote: Counting objects: 228734, done.
remote: Compressing objects: 100% (53705/53705), done.
remote: Total 228734 (delta 179855), reused 221914 (delta 174660)
Receiving objects: 100% (228734/228734), 58.45 MiB | 462 KiB/s, done.
Resolving deltas: 100% (179855/179855), done.

Before making any changes, create new local branch named msc-coder and switch to it.

$ git branch msc-coder
$ git checkout msc-coder
Switched to branch 'msc-coder'
$ git branch
  master
* msc-coder

Building process

Before we dive into codec implementation specifics lets try building FFmpeg. Go to root directory of FFmpeg distribution and execute configure script:

$ ./configure
...
Enabled decoders:
aac			cinepak			loco
...
Enabled encoders:
a64multi		h263p			pgm
...
Enabled demuxers:
aac			iff			pcm_u24be
...
Enabled muxers:
a64			ipod			pcm_mulaw
...

I have highlighted some important things in video codec development. First of all, we can see which encoders and decoders are currently enabled. Besides, we can also see enabled muxers/demuxers. Next steep is building with make. You can reduce build time by providing -j flag which distributes building process to multiple cores:

$ make -j5

After successful build we should get ffmpeg executable. Notice there is also ffmpeg_g which is compiled with debug symbols included. Check version of your build:

$ ./ffmpeg -version
ffmpeg version N-41414-g134d0f7
built on Jun  8 2012 16:48:32 with gcc 4.6.3
configuration: 
libavutil      51. 56.100 / 51. 56.100
libavcodec     54. 25.100 / 54. 25.100
libavformat    54.  6.101 / 54.  6.101
libavdevice    54.  0.100 / 54.  0.100
libavfilter     2. 78.101 /  2. 78.101
libswscale      2.  1.100 /  2.  1.100
libswresample   0. 15.100 /  0. 15.100

Libavcodec structures overview

Libavcodec library consists all audio/video codecs of FFmpeg. Developing new video codec requires knowing some basic structures. In this section I describe necessary structures which will be used to add new video codec.

AVCodec

Audio/Video codec structure defined in libavcodec/avcodec.h file. You can find much information in comments in this file. I took excerpt from this structure, describing only this fields which are used later.

typedef struct AVCodec {
    const char *name;
    const char *long_name;
    enum AVMediaType type;
    enum CodecId id;
    int capabilities;
    const AVRational *supported_framerates;
    const enum PixelFormat *pix_fmts;
    int priv_data_size;
    int (*init)(AVCodecContext *);
    int (*encode2)(AVCodecContext *avctx, AVPacket *avpkt, const AVFrame *frame, int *got_packet_ptr);
    int (*decode)(AVCodecContext *, void *outdata, int *outdata_size, AVPacket *avpkt);
    int (*close)(AVCodecContext *);
}

Fields description:

• const char *name - codec name. Should be unique among coders/decoders. Coder and decoder can have same name.
• const char *long_name - long, descriptive version of name specially for humans.
• enum AVMediaType type - Codec type based on AVMediaType enumeration.
• enum CodecId id - unique codec id defined CodecId enumeration.
• int capabilities - codec capabilities defined by bitmask of CODEC_CAP_* constans.
• const enum PixelFormat *pix_fmts - pixel formats supported by video codec.
• int priv_data_size - size in bytes of private codec context (library will allocate memory in priv_data field of AVCodecContext structure).
• int (*init)(AVCodecContext *) - function pointer to codec initialize procedure.
• int (*encode2)(AVCodecContext *avctx, AVPacket *avpkt, const AVFrame *frame, int *got_packet_ptr) - function pointer to encoding procedure. Output data should be written to AVPacket structure.
• int (*decode)(AVCodecContext *, void *outdata, int *outdata_size, AVPacket *avpkt) - function pointer to decoding procedure. structure.
• int (*close)(AVCodecContext *) - function pointer to codec close procedure (used for releasing resources).

AVFrame

Struct representing audio/video frame. Some fields are used for both audio and video encoding. I described here only field used in sample video codec. For more information refer to comments in source file.

typedef struct AVFrame {
	uint8t *data[AV NUM DATA POINTERS];
	int linesize[AV NUM DATA POINTERS];
	int width ,height;
	int key_frame;
	int coded_picture_number;
	int display_picture_number;
} AVFrame;

Fields description:

• uint8t *data[AV NUM DATA POINTERS] - pointers to specific channels of video frame.
• linesize[AV NUM DATA POINTERS] - size in bytes for each video frame channel.
• int width, height - width and height of video frame.
• int key_frame - flag indicating key frame (1 - yes, 0 - no).
• int coded_picture_number - picture number in bitstream.
• int display_picture_number - picture number in display.

AVPacket

Struct representing data packet which is muxed/demuxed. For more information refer to comments in source file.

typedef struct AVPacket {
	uint8t *data;
	int size;
	int flags;
	int duration;
} AVPacket;

Fields description:

• uint8t *data - pointer to packet data.
• int size - size in bytes of packet data.
• int flags - packet flags defined by bitmask of AV_PKT_FLAG constans..

AVCodecContext

Struct representing data packet which is muxed/demuxed. For more information refer to comments in source file.

typedef struct AVCodecContext {
	void * priv_data;
	int compression_level;
	int width, height;
	enum PixelFormat pix_fmt;
	int frame_number;
	int frame_bits;
} AVCodecContext;

Fields description:

• void * priv_data - Pointer to private codec context data. Size of context is defined by priv_data_size fild of AVCodecContext struct.
• int compression_level - compression level.
• int width, height - width and heigh of video frames.
• enum PixelFormat pix_fmt - Pixel format.
• int frame_number - Number of encoded/decoded video frames.
• int frame_bits - Number of bits used for the previously encoded frame.

Adding new video codec

In this section you can find instruction on adding simple codec stub with name msc. To add new codec you have to follow these steps:

1. Define codec id in enumeration CodecID in file libavcodec/avcodec.h. This value must be unique. For codec name msc, enum name should be CODEC_ID_MSC (CODEC_ID_XXX).

   
   enum CodecID {
   	...
   	CODEC_ID_MSC = 0x30000,
   	...
   };

2. Add codec source code file - libavcodec/msccoder.c. Video encoder and decoder are defined by AVCodec structure. Encoder variable should be defined with name ff_CODEC_NAME_encoder (for decoder variable name is ff_CODEC_NAME_decoder). We have to pass pointers to functions responsible for initialization, cleanup and encoding process. It is worth to define private coder context as struct, which can be used to store some data during coding process. Bellow is simple stub for encoder and decoder.

   
   #include "avcodec.h"
   #include "internal.h"
   #include "libavutil.h"
   
   typedef struct MscDecoderContext {
   	// decoder context
   } MscDecoderContext;
   
   typedef struct MscEncoderContext {
   	// encoder context
   } MscEncoderContext;
   
   static int decode_init(AVCodecContext *avctx) {
   	av_log(avctx, AV_LOG_INFO, "MSC decode init\n");
   
   	return 0;
   }
   
   
   static int decode(AVCodecContext * avctx, void *outdata, int *outdata_size, AVPacket *avpkt) {
   	av_log(avctx, AV_LOG_INFO, "MSC decode\n");
   
   	return 0;
   }
   
   static int decode_close(AVCodecContext *avctx) {
   	av_log(avctx, AV_LOG_INFO, "MSC decode close\n");
   
   	return 0;
   }
   
   
   static int encode_init(AVCodecContext *avctx) {
   	av_log(avctx, AV_LOG_INFO, "MSC encode init\n");
   
   	return 0;
   }
   
   
   static int encode_frame(AVCodecContext *avctx, AVPacket *avpkt, const AVFrame *frame, int *got_packet_ptr) {
   	av_log(avctx, AV_LOG_INFO, "MSC encode\n");
   
   	return 0;
   }
   
   static int encode_close(AVCodecContext *avctx) {
   	av_log(avctx, AV_LOG_INFO, "MSC encode close\n");
   
   	return 0;
   }
   
   AVCodec ff_msc_decoder = {
   	.name = "msc",
   	.type = AVMEDIA_TYPE_VIDEO,
   	.id = CODEC_ID_MSC,
   	.priv_data_size = sizeof(MscDecoderContext),
   	.init =	decode_init,
   	.close = decode_close,
   	.decode = decode,
   	.capabilities = CODEC_CAP_DR1,
   	.long_name = NULL_IF_CONFIG_SMALL("Msc coder"),
   };
   
   AVCodec ff_msc_encoder = {
   	.name = "msc",
   	.type = AVMEDIA_TYPE_VIDEO,
   	.id = CODEC_ID_MSC,
   	.priv_data_size = sizeof(MscEncoderContext),
   	.init = encode_init,
   	.close = encode_close,
   	.encode2 = encode_frame,
   	.pix_fmts =	(const enum PixelFormat[]) {PIX_FMT_YUV420P, PIX_FMT_NONE},
   	.long_name = NULL_IF_CONFIG_SMALL("Msc coder"),
   };

3. Register codec in file libavcodec/allcodecs.c in function avcodec_register_all(). We use specific MACROS for encoder/decoder adding in libavcodec. To register both encoder and decoder we can use REGISTER_ENCDEC macro. Note that CodecID and AVCodec variables should follow naming policy for this macro to work.

   
   void avcodec_register_all(void)
   {
   	...
   	REGISTER_ENCDEC  (MSC, msc);
   	...
   }

4. Now it is time to modify libavcodec Makefile. In encoders/decoders section of Makefile add this two lines. Note that CONFIG_MSC_DECODER and CONFIG_MSC_ENCODER will be automatically created.

   
   # decoders/encoders/hardware accelerators
   ...
   OBJS-$(CONFIG_MSC_DECODER)            += msccoder.o
   OBJS-$(CONFIG_MSC_ENCODER)            += msccoder.o
   ...
   

5. Before we build ffmpeg with make we have to invoke configure script once more. Without this we would get following error:

   
   libavcodec/allcodecs.c: In function ‘avcodec_register_all’:
   libavcodec/allcodecs.c:263:1: error: ‘CONFIG_MSC_ENCODER’ undeclared (first use in this function)
   libavcodec/allcodecs.c:263:1: note: each undeclared identifier is reported only once for each function it appears in
   libavcodec/allcodecs.c:263:1: error: ‘CONFIG_MSC_DECODER’ undeclared (first use in this function)

   Values CONFIG_MSC_ENCODER and CONFIG_MSC_DECODER are created by macro REGISTER_ENCDEC. In root directory executed configure script and build. Added codec should be visible as enabled in configure script output.

   
   $ ./configure
   ...
   Enabled decoders:
   ...
   adpcm_sbpro_2		exr			msc
   ...
   Enabled encoders:
   ...
   adpcm_ima_wav		msc			ra_144
   ...
   $ make -j5
   ...
   CC	libavcodec/msccoder.o
   ...
   

6. After successful build check if added codec is properly listed:

   $ ./ffmpeg -codecs
   ...
   Codecs:
    D...... = Decoding supported
    .E..... = Encoding supported
    ..V.... = Video codec
    ..A.... = Audio codec
    ..S.... = Subtitle codec
    ...S... = Supports draw_horiz_band
    ....D.. = Supports direct rendering method 1
    .....T. = Supports weird frame truncation
    ......F = Supports frame-based multi-threading
    ......S = Supports slice-based multi-threading
    ......B = Supports both frame-based and slice-based multi-threading
    -------
    ...
    DEV D   msc             Msc coder
    ...
   

Integration with muxers/demuxers

There are variety of existing audio/video containers in FFmpeg. You can write your own if you want. I used existing open source matroska container format. To add MSC codec as video codec to mkv container you need simply to add this line to libavformat/matroska.c file:

	const CodecTags ff_mkv_codec_tags[]={
	...
	{"V_MSC"            , CODEC_ID_MSC},
	...
	};

Then if you use msc coder with mkv container information about used codec will be automatically persisted. You can test it by simply compress video sequence and then check if ffmpeg is able to detect used codec by ffprobe utility.

$ ffmpeg -i resources/bowing_qcif.avi -vcodec msc results/bowing_qcif.mkv
...
Output #0, matroska, to 'results/bowing_qcif.mkv':
  Metadata:
    encoder         : Lavf54.6.101
    Stream #0:0: Video: msc, yuv420p, 176x144 [SAR 128:117 DAR 1408:1053], q=2-31, 200 kb/s, 1k tbn, 29.97 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> msc)
...
$ ffprobe results/bowing_qcif.mkv
...
Input #0, matroska,webm, from 'results/bowing_qcif.mkv':
  Metadata:
    ENCODER         : Lavf54.6.101
  Duration: 00:00:10.01, start: 0.000000, bitrate: 7821 kb/s
    Stream #0:0: Video: msc, yuv420p, 176x144, SAR 12:11 DAR 4:3, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
...

Creating patch

FFmpeg is used in various applications. Some of them use static linking with specific version of ffmpeg or even maintain their own repo for ffmpeg. Thus, best way to add new codec to them, is to prepare patch and then apply it. In following sections I am going to describe integration with MPlayer which uses static linking and integration with gstreamer-ffmpeg plugin which uses dynamic linking.

But first, lets create patch. If you followed my recommendation to create branch to work with new codec, you can use git diff option to create patch. Firstly commit all your current work to branch. Then you can crate msc_coder.patch file with redirect. You should be in root directory of ffmpeg.

$ git diff master..msc-coder > msc_coder.patch

To apply patch you simply invoke patch command with -p1 argument.

$ patch -p1 < msc_coder.patch

Integration with MPlayer

MPlayer uses static linking with FFmpeg libraries. You can invoke ./configure script which pulls new ffmpeg snapshot from git repository. Then you have to apply patches to add new codec. You can also create symbolic link with name ffmpeg in root directory of MPlayer distribution (Note that this solutions is not always feasible because of mismatch between version of ffmpeg that you have and version needed by mplayer).

To configure etc/codecs.conf you have to follow instructions from DOCS/tech/codecs.conf.txt of MPlayer source code distribution. In case of msc coder simply add these lines in video codecs section.

videocodec ffmsc
  info "FFmpeg MSC coder"
  status working
  fourcc MSC0 ; internal MPlayer FourCC
  driver ffmpeg
  dll msc
  out I420

You should also define internal MPlayer FourCC for added codec in file libmpdemux/mp_taglists.c.

	...
static const struct AVCodecTag mp_codecid_override_tags[] = {
	...
	{ CODEC_ID_MSC,            	  MKTAG('M', 'S', 'C', '0')},
	...

After applying these changes you can build mplayer and install it. On Ubuntu 12.04 xorg-dev package is required for proper building of self compiled mplayer.

$ ./configure
$ make -j5
$ sudo make install
$ mplayer results/bowing_qcif.mkv

Integration with gstreamer-ffmpeg plugin

Ubuntu 12.04 gstreamer-ffmpeg plugin uses fork of ffmpeg - libav. By the time of writing, differences between ffmpeg and libav were so small, that patching libav with patch prepared for ffmpeg caused no problems. GStreamer is used by various applications including default video movie player for GNOME - Totem or thumbnails preview in Nautilius file manager. I am going to describe steps necessary to get thumbnails preview in nautilius file manager using ffmpegthumbnailer.

By default nautilius is using totem-video-thumbnailer as a video thumbnailer in GNOME desktop environment. To make it work with ffmpeg you can use gstreamer-ffmpeg plugin and ffmpegthumbnailer. In Ubuntu gstreamer-ffmpeg package is compiled with system provided version of libav. So first step is to patch libav for whole system and install it. On Ubuntu you can download source packages with apt-get command with source argument.

$ sudo apt-get source libav-source

You have to pull all required build-dependent packages before building deb package. You can do this with apt-get build-dep command.

$ sudo apt-get build-dep libav-source

You should get directory libav-VERSION with libav framework. As mentioned before you can try to simply patch source with patch created for ffmpeg. Note that deb source packages provide their own patches in .pc/ directory and they can interfere with your patches. Please refer to Ubuntu Packaging Guide for more information how you should apply your own patches. To build binary package use dpkg-buildpackage commnand within libav-VERSION directory.

$ dpkg-buildpackage -rfakeroot -uc -b

After successful build you should get deb packages in parent directory. Install them with dpkg command.

$ ls *.deb
ffmpeg_0.8.3-0ubuntu0.12.04.1_all.deb           libavdevice-dev_0.8.3-0ubuntu0.12.04.1_i386.deb  libav-tools_0.8.3-0ubuntu0.12.04.1_i386.deb
ffmpeg-dbg_0.8.3-0ubuntu0.12.04.1_i386.deb      libav-doc_0.8.3-0ubuntu0.12.04.1_all.deb         libavutil51_0.8.3-0ubuntu0.12.04.1_i386.deb
ffmpeg-doc_0.8.3-0ubuntu0.12.04.1_all.deb       libavfilter2_0.8.3-0ubuntu0.12.04.1_i386.deb     libavutil-dev_0.8.3-0ubuntu0.12.04.1_i386.deb
libavcodec53_0.8.3-0ubuntu0.12.04.1_i386.deb    libavfilter-dev_0.8.3-0ubuntu0.12.04.1_i386.deb  libpostproc52_0.8.3-0ubuntu0.12.04.1_i386.deb
libavcodec-dev_0.8.3-0ubuntu0.12.04.1_i386.deb  libavformat53_0.8.3-0ubuntu0.12.04.1_i386.deb    libpostproc-dev_0.8.3-0ubuntu0.12.04.1_i386.deb
libav-dbg_0.8.3-0ubuntu0.12.04.1_i386.deb       libavformat-dev_0.8.3-0ubuntu0.12.04.1_i386.deb  libswscale2_0.8.3-0ubuntu0.12.04.1_i386.deb
libavdevice53_0.8.3-0ubuntu0.12.04.1_i386.deb   libav-source_0.8.3-0ubuntu0.12.04.1_all.deb      libswscale-dev_0.8.3-0ubuntu0.12.04.1_i386.deb
$ sudo dpkg --install *.deb

Then you can check that we are using libav ffmpeg fork (look for line containing information Copyright (c) 2000-2012 the Libav developers).

$ ffmpeg -version
ffmpeg version 0.8.3-4:0.8.3-0ubuntu0.12.04.1, Copyright (c) 2000-2012 the Libav developers
  built on Jul 15 2012 18:19:18 with gcc 4.6.3
*** THIS PROGRAM IS DEPRECATED ***
This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.
ffmpeg 0.8.3-4:0.8.3-0ubuntu0.12.04.1
libavutil    51. 22. 1 / 51. 22. 1
libavcodec   53. 35. 0 / 53. 35. 0
libavformat  53. 21. 0 / 53. 21. 0
libavdevice  53.  2. 0 / 53.  2. 0
libavfilter   2. 15. 0 /  2. 15. 0
libswscale    2.  1. 0 /  2.  1. 0
libpostproc  52.  0. 0 / 52.  0. 0

We are now ready to build gstreamer-ffmpeg plugin. As earlier we are going to create patched .deb package version. You can find information about patching policy for this project in HACKING file from git source version. Let's firstly download source package and build dependencies.

$ sudo apt-get source gstreamer0.10-ffmpeg
$ sudo apt-get build-dep gstreamer0.10-ffmpeg

To integrate our new codec with gstreamer we have to modify gst_ffmpeg_codecid_to_caps function ext/ffmpeg/gstffmpegcodecmap.c file. It's purpose is to convert provided codec_id and AVCodecContext to GstCaps caps structure.

GstCaps *
gst_ffmpeg_codecid_to_caps (enum CodecID codec_id,
	AVCodecContext * context, gboolean encode)
	{
	...
	switch (codec_id) {
	...
	case CODEC_ID_MSC:
	caps = gst_ff_vid_caps_new (context, codec_id, "video/x-msc", NULL);
 	break;
 	...

After applying specified patch we can build and install gstreamer0.10-ffmpeg package. Our codec should be visible in output of gst-inspect ffmpeg command.

$ gst-inspect ffmpeg
...
ffdec_msc: FFmpeg Msc coder decoder
...

Now we are ready to replace totem-video-thumbnailer with ffmpegthumbnailer. Firstly install ffmpegthumbnailer package.

$ sudo apt-get install ffmpegthumbnailer

Then you have to replace following contents of /usr/share/thumbnailers/totem.thumbnailer:

[Thumbnailer Entry]
TryExec=/usr/bin/totem-video-thumbnailer
Exec=/usr/bin/totem-video-thumbnailer -s %s %u %o
MimeType=application/mxf;application/ogg;...

with this (note usage of video/x-msc mime type used in ext/ffmpeg/gstffmpegcodecmap.c file):

[Thumbnailer Entry]
TryExec=ffmpegthumbnailer
Exec=ffmpegthumbnailer -s %s -i %i -o %o -c png -f -t 10
MimeType=application/mxf;application/ogg;...;video/x-msc;

Then just quit nautilius and clear thumbnails cache

nautilus -q
rm ~/.thumbnails/fail/gnome-thumbnail-factory/*
rm ~/.thumbnails/normal/*

Now you should get thumbnail preview of video coded by msc-coder codec in mkv container.

$ pwd
~/private/msc/msc-coder/results
$ ffprobe akiyo_qcif.mkv
...
Stream #0.0: Video: msc, yuv420p, 176x144, PAR 12:11 DAR 4:3, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
...