Fluggo Media Library¶
The Fluggo media library is for high-quality processing of audio and video for use in non-linear editing and compositing systems. It's written primarily for use in Python.
Quick links: Building the library :: Using video sources in Python :: Using video sources in C :: Writing video sources :: Library reference
What it can do now¶
Right now, using the library, you can:
- Decode DV video into 4:4:4 linear RGB with correct subsample reconstruction using the FFVideoSource
- Remove 2:3 pulldown with the Pulldown23RemovalFilter
- Blend video with the VideoMixFilter
- Construct sequences of video with the VideoSequence class
- Composite complex arrangements of video clips over time with the Workspace class
- If your video hardware supports it, perform most operations in hardware using OpenGL and GLSL
- Display video in OpenGL widgets for GTK and QT, synced to either the system clock or the AlsaPlayer
- Decode simple audio streams using the FFAudioSource
- Play sound through ALSA using the AlsaPlayer class
- Animate some properties using frame functions
- Write DV video and sound to an AVI file (not tested in awhile, but no reason it shouldn't still work)
- Easily write your own video sources/filters, audio sources/filters, frame functions, and clock sources
- Scale video using FIR filters and animatable properties (progressive only, interlaced to come later) using the VideoScaler
- Produce thumbnails for videos on a background thread using the VideoPullQueue
Medium-term goals include everything needed (in processing) to make a basic, usable video editor, namely:
- Mixing audio in the Workspace class
- Basic color correction filter
- OpenGL support for all video operations
- Hints to make certain operations faster
Example¶
It's easy to build chains of filters:
from fluggo.media import process
from fluggo.media.basetypes import *
videro = process.FFVideoSource('softboiled01;03;21;24.avi')
pulldown = process.Pulldown23RemovalFilter(videro, 0)
color = SolidColorVideoSource(rgba(1.0, 0.0, 0.0, 0.5), box2i(50, 50, 100, 100))
mix = process.VideoMixFilter(src_a=pulldown, src_b=color,
mix_b=process.LinearFrameFunc(a=1/300.0, b=0))
In fact, it's even easier than building a GUI to display it:
from fluggo.media import qt from PyQt4.QtCore import * from PyQt4.QtGui import * from PyQt4.QtOpenGL import * import sys app = QApplication(sys.argv) clock = process.SystemPresentationClock() format = QGLFormat() video_widget = qt.VideoWidget(format) video_widget.setDisplayWindow(box2i(0, -1, 719, 478)) video_widget.setPixelAspectRatio(640.0/704.0) video_widget.setPresentationClock(clock) video_widget.setVideoSource(mix) video_widget.show() clock.play(1) quit(app.exec_())
Overall goals¶
After reading and comparing other frameworks, I decided on these goals:
- Produce high-quality images and sound
- Make writing video and audio filters easy
- Keep video and audio paths absolutely separate
- Use as little metadata as possible
- Make it scriptable in Python
High quality. Fluggo Media pays attention to the video standards. I've seen for-pay solutions that get it wrong, and I know that if I pay attention to these details, others do, too.
(Insert example here)
Fluggo Media also lets you override all of the settings. When you decode a video file, you'll be able to determine the transfer function, reconstruction filters, and even the color matrix.
Easy coding. There are no pins and pads, no negotiating the type of video data or who is going to allocate the buffers. Video sources can be used more than once, because there are no connections between filters, just calls upstream. Setting a source is as simple as setting a pointer.
All video is 4:4:4 RGBA floating-point linear by the time it gets to you. The only difference is whether this is stored as 16-bit float, 32-bit float, or in an OpenGL surface. You can write a filter to handle all three, but you only need to write the 16-bit or 32-bit function to get a working filter. The upstream-fetching functions handle translations between all three automatically.
Separate video and audio. No audio in my video filter, thanks. I don't care if they are synced. We go to the extreme here: Fluggo Media doesn't even decode them together, even if both streams are in the same file. The filter chains are entirely separate. The only thing that ties them together is the presentation clock, which makes sure they play in sync.
No metadata. Video filters don't know a frame's:
- Frame rate
- Aspect ratio
- Interlacing
- Color primaries
Video filters don't need to know these things, because video filters shouldn't be making metadata decisions. Even if a file does say what its aspect ratio is, who's to say it's right? Provide the metadata to the calling program and let it decide what to do. Fluggo video filters only know what the size of the picture is (in pixels) and its position in the final frame. (link to coordinate system)
Python. I love Python, and I love the idea of being able to script a heavy-duty video processing machine. Python is Fluggo Media's primary interface, and I plan to keep it that way, but for those of you who want to use it in C or wrap it in your language of choice, I'm working on making a C interface as well.
(You might ask - how do you get performance out of Python? The answer is that I don't. All the video processing takes place in extension modules. Python is the interface for setting up filter chains and user interfaces.)
Compared to other frameworks¶
I evaluated other frameworks to use for my editor and came up short. The two I evaluated were GStreamer (and GNonLin) and the current favorite for open-source editors, MLT.
Video formats. GStreamer and MLT both pass specific video data, such as 8-bit YCbCr 4:2:2, through the stream. GStreamer is more pedantic about this.
There are a few problems with this:
- Each filter has to deal with every kind of video format the framework supports (e.g. 8-bit/10-bit/12-bit/whatever, RGB/RGBA/YCbCr, 4:1:1/4:2:2/4:2:0/4:4:4, in any combination)
- The filters don't have enough info to work with each format successfully (Where are your chroma subsamples located in your 4:2:0 video?)
- Making the filters work with each format successfully, and each new format successfully, is a lot of work
I don't like doing a lot of work, so I standardized on a format that can handle any video out there.
Metadata.
Video and audio.