Using Stable Diffision models for Colorization

[safshe]

As a natural extension of the "Fix Image" tab, I've added the new "Fix Video" tab.

Suppose you've already colorized a clip with CMNET2 and DitServerRPC. There will be a folder called "ref_qwen" containing the colored frames.
Suppose you've viewed the clip and want to change the color of some frames. Let's say these frames range from #34 to #525. After fixing them with the "Fix Image" tab, you can recolor the video only in the identified frame range (in this case, from #34 to #525). As shown in the image below.


I hope this helps; in some cases, it was necessary for me (which is why I added the implementation).

Since for technical reasons, I still need to re-encode the entire clip, it's mandatory to use as input the clip already colorized and to have NVEnC installed as explained in the README.

Dan

Please add a new feature to the Fix Image pipeline.
Proposed Workflow:

• Step 1: Load a pre-colorized image to act as the primary color base.
  
• Step 2: Select a new reference image that you want to colorize.
  
• Step 3: Perform the colorization on the reference image, but strictly guide the colorization based on the palette and tones of the already loaded colorized frame.
  
  

I have tested a similar workflow in ChatGPT, and the logic works well. I hope we can implement this

[Dan64]

Hi safshe,

   you must try with ComfyUi. 
   If it works with ComfyUi there is some possibility that it could be added in the GUI of DiTServerRPC.

Dan

[hallomanbh]

While the model successfully colorizes and outputs all images, maintaining consistency across frames remains a major challenge. In many cases, the colors in the final colorized frame differ from those in the reference frame. As a result, the same shot may appear with varying colors across consecutive frames, leading to noticeable temporal inconsistencies and flickering in the video.

[Dan64]

If you extract a reference frame every 30-40 frames, flickering is very unlikely and this also improves color consistency.

At the end of reference frame's colorization I usually spend some time in checking the colorized frames in folder "ref_qwen".
In most cases it is just enough to delete the frames with different colors. Only in the the case are missing adjacent frames with correct colors, I'm using "Fix Image" to fix the color (this represent about the 10% of total frames deleted/removed).

In any case in Hybrid there are a lot of good filters to remove the flickering you can apply them at the end of colorization process.

Dan

[Dan64]

About the paths,...  Due to the changes in R75+ current Hybrid does load plugins from "Vapoursynth/Lib/site-packages/vapoursynth/plugins2" is they are available through pip. Main gain from this is that, Vapoursynth will, if the a dll has multiple versions (like for example zsmooth, vszip, hysteresis, cranexpr), automatically load the best suited when loading the base .dll (i.e. zsmooth.dll).
(I install/update the plugins through pip and move them to the plugins2 folder, to avoid autoloading.)
=> I plan to remove dlls which are in the plugins2 folder from vsfilters in the (near) future. (atm. they are still there)
Not sure whether I will switch to autoloading dlls in the future, but atm. that is not planned, so plugins2 folder is likely to stay for now.

side note: the next problematic thing will happen when Vapoursynth drops (atm. you just get annoying warning messages) API3 support,...
(hopefully this will not happen as fast as new Vapoursynth versions come out nowadays,...)

Cu Selur

I noted that some of plugins used by HAVC (see post: #44) are available now as plugins via pip.
The missing ones are:

1) TCanny (built using APIv4)
6) Retinex  (built using APIv3)
7) SCDetect  (built using APIv3)
8) ReduceFlicker  (built using APIv3) 
9) LSMASHSource  (built using APIv3) 

Regarding SCDetect, I noted that is available this plugin: vapoursynth-scenechange
Do you know it ?, do you think that can be a good replacement for SCDetect ?

How these plugings, installed via pip, can be checked and loaded inside a script ? can you provide an example ?

Another final question: I noted that the only plugin to reading clip/movies installed via pip is vapoursynth-bestsource, do you know if there are other APIv4 plugins that can be used in substitution of bestsource (still too slow on big movies)  ?

Dan

[Selur]

Plugins I install through pip (python -m pip install -U -r plugins.txt):

vapoursynth vapoursynth-adaptivegrain vapoursynth-akarin vapoursynth-awarp vapoursynth-bestsource vapoursynth-bilateralgpu vapoursynth-bm3d vapoursynth-bwdif vapoursynth-cranexpr vapoursynth-d2vsource vapoursynth-dctfilter vapoursynth-deblock vapoursynth-dedot vapoursynth-descale vapoursynth-descratch vapoursynth-dghdrtosdr vapoursynth-dotkill vapoursynth-edgefixer vapoursynth-edgemasks vapoursynth-eedi3 vapoursynth-fillborders vapoursynth-fmtconv vapoursynth-hysteresis vapoursynth-iscombed vapoursynth-knlmeanscl vapoursynth-mvtools vapoursynth-nlm-cuda vapoursynth-nlm-ispc vapoursynth-resize2 vapoursynth-sangnom vapoursynth-scenechange vapoursynth-sneedif vapoursynth-subtext vapoursynth-timecube vapoursynth-vivtc vapoursynth-vszip vapoursynth-wnnm vapoursynth-zit vapoursynth-znedi3 vapoursynth-zsmooth vs-placebo vsnoise

and move them (from "Vapoursynth\Lib\site-packages\vapoursynth\plugins") into "Vapoursynth\Lib\site-packages\vapoursynth\plugins2" to avoid automatic loading.
here's the content of the plugins2 folder:

Vapoursynth\Lib\site-packages\vapoursynth\plugins2>ls -R .: DGHDRtoSDR.dll bilateralgpu deblock.dll eedi3m.dll libdescale.dll sneedif vszip KNLMeansCL.dll bm3d.dll dedot.dll fillborders.dll libvs_placebo.dll subtext.dll wnnm.dll adaptivegrain bwdif.dll descratch.dll fmtconv.dll mvtools.dll timecube.dll zit.dll addnoise.dll cranexpr dotkill.dll hysteresis nnedi3_weights.bin vivtc.dll znedi3.dll akarin d2vsource.dll edgefixer.dll iscombed.dll resize2.dll vsnlm_cuda.dll zsmooth awarp.dll dctfilter.dll edgemasks.dll libbestsource.dll sangnom.dll vsnlm_ispc.dll '.\adaptivegrain': adaptivegrain_rs.avx2.dll adaptivegrain_rs.dll adaptivegrain_rs.zn4.dll manifest.vs '.\akarin': libakarin.dll libzstd.dll '.\bilateralgpu': bilateralgpu.dll bilateralgpu_rtc.dll '.\cranexpr': cranexpr.avx2.dll cranexpr.dll cranexpr.zn4.dll manifest.vs '.\hysteresis': hysteresis.dll hysteresis.v2.dll hysteresis.v3.dll manifest.vs '.\sneedif': libsneedif.dll nnedi3_weights.bin '.\vszip': manifest.vs vszip.avx2.dll vszip.dll vszip.zn4.dll '.\zsmooth': manifest.vs zsmooth.avx2.dll zsmooth.dll zsmooth.zn4.dll

Do you know it ?, do you think that can be a good replacement for SCDetect ?

Works fine here. (is used by Hybrid, vsscripts are adjusted for it)
I always call misc.SCDetect, which uses a plugin if it's available and falls back to a manual scene change detection otherwise.

Another final question: I noted that the only plugin to reading clip/movies installed via pip is vapoursynth-bestsource, do you know if there are other APIv4 plugins that can be used in substitution of bestsource (still too slow on big movies) ?

DGDecNV depending on the source. Aside from BestSource all the other libav based source filters are still APIv3 since nobody cared enough to update them to APIv4.


Cu Selur