Pompei¶

Installation¶

If you have PIP installed:

(sudo) pip install ez_setup pompei

Or unzip the source code in a directory and type in a terminal:

sudo python setup.py install

Pompei depends on MoviePy which will be automatically installed during the installation of Pompei.

How it works¶

The mosaic algorithm proceeds as follows:

• Many frames of the movie (for instance one every 5 second) are extracted and downsized to make small thumbails. This takes a few minutes, but you will only need to do it once for each movie. The remaining steps takes less than 30 seconds in total.
• The image from which the mosaic will be made is divided into small rectangular regions.
• At first, each region is matched with the thumbnail that fits it best. To speed up computations Pompei reduces each subregion of the image, and each thumbnail extracted in step 1, to a few average pixels (for instance, 3x4 pixels), called the “signature” of the image. An image region and a thumbnail are said to “fit” well when there signatures are similar.
• The thumbnails associated with some regions are then changed to ensure that no thumbnail is represented too many times (first pass), and that thumbnails whitch are near in time in the original movie are well spread apart in the final picture (second pass).
• Finally all the thumbnails are assembled on a big canvas to form the mosaic, which is written to a file.

Some pictures can be well mosaiced using the frames of a given movie, but some can’t. To help you choose the “right” image for your mosaic, Pompei provides a method (depending on the package Scikit-Learn) to find which thumbnail image extracted from the movie during the first step can be well reconstructed with the other thumbnails. This is not fully tested but seems to work quite well, here is the algorithm:

• Pool together the colors of the different images, and find the K=10 colors which represent best the general colors of the movie, using a K-means clustering algorithm.
• For each thumbnail, replace each of its pixels by the nearest of the K selected colors. If the color change didn’t modify much the thumbnail, it means that that the color in this frame are well represented in the movie in general, and a mosaic could be made from this image. Therefore the difference between the original thumbnail and its color-quanitzed version gives you a score. Frames with the lowest score are the best candidates for a mosaic.

Example of code¶

In this script we extract frames from a movie and detect 50 frames which could make a good mosaic, then we select one particular frame and turn in into a mosaic.

from pompei import MovieFrames

# The next command is only used once to extract frames from the movie.

movieframes = MovieFrames.from_movie('gladiator.flv', # a video file
    foldername='gladiator', # directory where frames are extracted
    fps=1.0/5, # take one frame every 5 seconds.
    crop=(10*60,10*60), # cut out first and last 10 min.
    sig_dim=(3,3)) # frames signatures: 3x3 pixels

# For when the movie frames are already extracted:

movieframes = MovieFrames(foldername='gladiator')


# (optional) find the thumbnails that would make great mosaics.
# These lines require the Python package Scikit-learn.
# The thumbnails are extracted and saved with the corresponding time and
# score in the title, so that the original image can easily be retrieved

for i, score in movieframes.find_mosaicable_frames()[:50]:
    time = movieframes.times[i]
    movieframes.extract(i, "gladiator_%d_%.2f.png"%(time,score))



# Load the image that you are going to turn into a mosaic, either
# from an image file or a frame from a video file

from moviepy.editor import VideoFileClip, ImageClip
image = ImageClip("some_image.jpeg") # or next line:
image = VideoFileClip("gladiator.flv").get_frame('01:26:43')


# Make a mosaic from the frame. Many options available.

movieframes.make_mosaic( image,
  "gladiator_mosaic.png", # output file (will be large and heavy !)
  frames_in_width=60, # 'image' is reconstructed with 60x60 thumbails
  max_occurences=10, # wished maximal occurence of a thumbnail.
  maxiter=3000,     # abandomn optimization after 3000 steps
  spatial_window=2, # frames at a distance of 2 or less must be
  time_window=3)    # separated by 3 indices (here 15 movie seconds)

Command line API¶

Not yet implemented, these are just thoughts. When Pompei is installed it also provides the following command line interface:

>>> pompei.py -h #to get detailed help
>>> pompei.py video_to_folder my_video.mp4 my_folder --signatures=3x3
>>> pompei.py find_best_frames my_folder --lum_min=50 --nframes=50
>>> pompei.py extract_image my_video.mp4 '01:02:30.10' myframe.png
>>> pompei.py make_mosaic folder myframe --res=50 --max_occurences=12

Customizing Pompei¶

There are many parameters you can tweak in Pompei, and the code is very modular, which makes it easy to change any part of the algorithm. The simplest way to do this it to first go see in the module’s code which method performs the task that you wish to customize, then, in your script, overwrite this method, for instance by creating a subclass of MovieFrames:

class MovieFrames2(MovieFrames):
    def _find_best_matches(self, a, b):
        # write your own version of this function

Reference manual¶

class pompei.MovieFrames(folder)¶

Base class in Pompei. Objects represent a series of thumbnail, which are produced from a movie with MovieFrames.from_movie(), are analyzed with MovieFrame.find_mosaicable_frames(), and are used to reconstitute mosaics with MovieFrames.make_mosaic().

Methods

find_mosaicable_frames(nclusters=8, luminosity_threshold=50)¶

Finds the frames whose colors resemble most the main colors of the movie, and thus would make good candidates for a mosaic. This is highly experimental.

Requires Scikit-learn installed.

Returns [(frame1,score1), (frame2, score2)...] where the lowest scores indicates frames better suited for a mosaic.

static from_movie(filename, foldername=None, tt=None, fps=None, crop=(0, 0), thumbnails_width=120, sig_dim=(2, 2))¶

Extracts frames from a movie and turns them into thumbnails.

Parameters:

filename : Name of the legally obtained video file (batman.avi, superman.mp4, etc.) foldername : The extracted frames and more infos will be stored in that directory. tt : An array of times [t1, t2...] where to extract the frames. Optional if crop : Number of seconds to crop at the beginning and the end of the video, to avoid opening and en credits. (seconds_cropped_at_the beginning, seconds_cropped_at_the_end) thumbnails_width : Width in pixels of the thumbnails obtained by resizing the frames of the movie. sig_dim : Number of pixels to consider when reducing the frames and thumbnails to simple (representative )signatures. sid_dim=(3,2) means 3x2 (WxH) pixels.

make_mosaic(image, outputfile, frames_in_width=50, max_occurences='auto', maxiter=500, time_window=2, spatial_window=2, matching_quality=0.7)¶

Makes a mosaic file from the best_matches found.

Finds the right frames to reconstitute the given image, in three steps:

1. Find the optimal frame for each subregion of the given image.
2. Change the frames so as to avoid the same frames being used too many

times.

3. Change the frames so that frames that are near in time in the original

movie will not appear near from each other in the final mosaic.

Parameters:

image : The (RGB WxHx3 numpy array) image to reconstitute. frames_in_width : How many frames are in the width of the picture (determines the ‘resolution’ of the final mosaic) max_occurences : How many occurences of the same frame are allowed in step 2. If auto, this is not taken into account and the algorithm will try and reduce the number of occurences until it reaches maxiter iterations or bad overall matching quality (see below) maxiter : Number of iterations in step 2 when reducing the number of occurence of the most frequent frames. matching_quality : The program will stop when the current total matching error is less than (initial_score / matching_quality). this is to avoid that the program degrades the quality of the mosaic too much. time_window : The frames will be changed in step 3 so