Week 3: Merging tracks and coding

Mar 26, 2021

Hello everyone, welcome to the week 3 of my internship. After getting the
first audio from the plugin chain and understanding the overall algorithm of
my plugin chain (see the images from last week), it is now to make some progress and optimization. In my blog
below, I will illustrate how I process the images with more complicated
algorithms and how I start to adding code to implement new algorithms.

Combining different tracks

From the last section, we could see that there are different color filters to
change the color of the scanned image. Combining those images together with
tracking would decrease the noise in the produced music.

In order to add the produced tracking to a different image, we have to use the
plugin `LoadTrack`, which loads the track to the image. Then, we can use the
plugin `ShiftTrack` to find the appropriate distance the tracking should
shift. Finally, we can use the plugin `WriteWav` to produce the music.

The overall procedure could be shown below


(Clarification: this diagram is created by my mentor Mr. Peter Alyea, some unessential plugins are emitted.)

As shown in the image above, we first use the red filter to filter the scanned
image and then get the tracking. Then, we filter the same scanned image using
the blue filter and the green filter. For the images produced by the blue and
green filters, we will then load the track produced on the red filtered image.
Finally, we can retrieve the audio from the three images and combine them
together using the software Audacity. When using Audacity, we can use the
option `mix` to merge the three audios together.

When dealing with the tracking on the scanned image, it is noteworthy to look
at the option at the plugin `ShiftTrack`. In this plugin, a positive number
would mean shifting the tracking horizontally to the right while a negative
number means shifting the tracking to the left.

Working on my first plugin: CombineTrackWeighted

From the workflow introduced in the last section, we could see the mechanism
is overall using the same tracking on different images. Another mechanism is
envisioned: what if we combine different tracks and process them on the same
image? A plugin like this, CombineTrack, has been written by the programmers
of Weaver. However, there are ways to make it better: when some tracking are
way astray from the groove in the scanned image, we should make the weight of
the tracking much smaller than those of the other. Besides, we should also
look for other interpolation ways and use other factors, such as brightness of
the scanned image and the tracking, to produce a better merge of the tracking.

The data structure we will deal with when writing this plugin in is the `track
(List<List>)`, which is a list of lists in the C# programming language. In
particular, it has two dimensions, and a particular item could be accessed by
using the index i, j and `track[i][j]`. We also have the following

– track[i][0] is the vertical position of the track.
– track[i][1] is a placeholder of the merged track.
– the later 6 indexes store the stereo audios, with three left and three
– information such as brightness or width are stored in the later indexes.

With those information, I am ready to start working on adding weights to the
`CombineTrack` plugin.

Next Week

1. Sharing my progress on writing the plugin.
2. Sharing my progress on using the filter to eliminate the ‘ghosting light’

One Reply to “Week 3: Merging tracks and coding”

  1. Eric M. says:

    This is all very interesting stuff! Is it possible for you to show us some of the images you are working with? I’d love to see how you manipulate it!

Leave a Reply