{"id":10827,"date":"2023-11-29T09:25:02","date_gmt":"2023-11-28T23:25:02","guid":{"rendered":"https:\/\/weirdwonderfulai.art\/?p=10827"},"modified":"2023-11-29T16:51:04","modified_gmt":"2023-11-29T06:51:04","slug":"stable-video-diffusion-using-comfyui","status":"publish","type":"post","link":"https:\/\/weirdwonderfulai.art\/comfyui\/stable-video-diffusion-using-comfyui\/","title":{"rendered":"Stable Video Diffusion using ComfyUI"},"content":{"rendered":"\n<p>Last week&#8217;s hottest news other than the drama unfolding at Open.AI with Sam, this was a very positive news when Stability.AI announced that they have released <a href=\"https:\/\/stability.ai\/news\/stable-video-diffusion-open-ai-video-model\" target=\"_blank\" rel=\"noreferrer noopener\">Stable Video Diffusion<\/a> (21st November 2023). A few days later the first two models are available for experimental\/evaluation use (ie. Non-commercial), so of course then we wait for the community to take this to next level and make it usable on your own PC.<\/p>\n\n\n\n<p>We have two models available:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/huggingface.co\/stabilityai\/stable-video-diffusion-img2vid\" target=\"_blank\" rel=\"noreferrer noopener\">Stable Video Diffusion<\/a> &#8211; As its referred to as SVD, its able to produce short video clips from an image at 14 frames at resolution of 576&#215;1024 or 1024&#215;574. This is sufficient for small clips but these will be choppy due to the lower frame rate. Model file is svd.safetensors 9.56GB<\/li>\n\n\n\n<li><a href=\"https:\/\/huggingface.co\/stabilityai\/stable-video-diffusion-img2vid-xt\" target=\"_blank\" rel=\"noreferrer noopener\">Stable Video Diffusion XT<\/a> &#8211; SVD XT is able to produce 25 frames at resolution of 576&#215;1024 or 1024&#215;574. This would provide smoother results. Model file is svd_xt.safetensors 9.56GB<\/li>\n<\/ul>\n\n\n\n<p>Here are some samples available and shared at the above links. These are smooth GIF images.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/svd-sample.gif\" alt=\"\" class=\"wp-image-10828\"\/><\/figure>\n\n\n\n<p>Now that you have the links to the models you can download the model file (.safetensors) which is about ~10 GB, depending upon what you prefer to use. I will be using Stable Video Diffusion XT version to push my <a href=\"https:\/\/weirdwonderfulai.art\/general\/building-a-custom-ai-art-generation-pc\/\" data-type=\"post\" data-id=\"5013\">RTX4080 16GB card<\/a>.<\/p>\n\n\n\n<p>Next we need the ComfyUI workflow which is available for <a href=\"https:\/\/comfyanonymous.github.io\/ComfyUI_examples\/video\/\">download<\/a>. There is one workflow for Text-to-Image-to-Video and another for Image-to-Video.<\/p>\n\n\n\n<p>I am going to experiment with Image-to-Video which I am further modifying to produce MP4 videos or GIF images using the Video Combine node included in <a href=\"https:\/\/github.com\/Kosinkadink\/ComfyUI-VideoHelperSuite\" target=\"_blank\" rel=\"noreferrer noopener\">ComfyUI-VideoHelperSuite<\/a><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"410\" src=\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD-Workflow-sample-1024x410.jpg\" alt=\"\" class=\"wp-image-10832\" srcset=\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD-Workflow-sample-980x392.jpg 980w, https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD-Workflow-sample-480x192.jpg 480w\" sizes=\"auto, (min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw\" \/><figcaption class=\"wp-element-caption\">This a preview of the workflow &#8211; download workflow below<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Download ComfyUI Workflow<\/h2>\n\n\n<a  data-e-Disable-Page-Transition=\"true\" class=\"download-link\" title=\"\" href=\"https:\/\/weirdwonderfulai.art\/download\/svd-image-to-video\/?tmstv=1731662614\" rel=\"nofollow\" id=\"download-link-10834\" >\n\tSVD-Image-to-Video\t(31131 downloads\t)\n<\/a>\n\n\n\n\n<p>The first experiment is using a frame that I have a frame from another project I&#8217;ve been working on. The result was pretty cool using the default settings of the workflow. The camera movement is pretty cool as the subject moves the camera is following her.<\/p>\n\n\n\n<p>This result was produced in 52 seconds in my setup.<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video controls src=\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD_XT_00001.mp4\"><\/video><\/figure>\n\n\n\n<p>Now when I&#8217;m experimenting further with 24fps and producing 2-3 seconds worth of video it takes 2-3 minutes. This is pretty acceptable at 1024 by 576 pixels, 48 video frames, 24 fps &#8211; this gives me 3 second clip.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"183\" src=\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD-generation-time-1024x183.jpg\" alt=\"\" class=\"wp-image-10831\" srcset=\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD-generation-time-1024x183.jpg 1024w, https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD-generation-time-980x175.jpg 980w, https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD-generation-time-480x86.jpg 480w\" sizes=\"auto, (min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw\" \/><\/figure>\n\n\n\n<p>The key parameters of SVD_img2vid_Conditioning are:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>video_frames &#8211; the number of frames that will be generated. You have to calculate this to align with your fps.<\/li>\n\n\n\n<li>motion_bucket_id &#8211; the number dictates how much motion will be in the video. At the moment this is only way to control the movement, there are no prompts or other controls to guide the movement. It&#8217;s fun to experiment with these.<\/li>\n\n\n\n<li>fps &#8211; frames per second, higher the value smoother the video and smaller value the choppy the video will be. As you start I suggest start with 6 fps and then can move up to 24\/25fps if your GPU can handle it.<\/li>\n\n\n\n<li>augmentation_level &#8211; the amount of noise that will be added to your initial image, lower value means the video will look a lot like original image and higher values the more it will move away from original image.<\/li>\n<\/ul>\n\n\n\n<p><strong>Tip:<\/strong> If you get this error (below) when running the workflow simply uninstall or disable FreeU.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Error occurred when executing KSampler:\nunsupported operand type(s) for *=: 'int' and 'NoneType'<\/code><\/pre>\n\n\n\n<p>All my experiments have been interesting and some produced good results but nothing like the preview videos shared by Stability.AI above. I guess their secret recipe is not shared yet but it would be wonderful to know the settings they were using so we can also replicate them.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-1 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-video\"><video controls src=\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD-WW.mp4\"><\/video><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-video\"><video controls src=\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/JellyFish_SVD.mp4\"><\/video><\/figure>\n<\/div>\n<\/div>\n\n\n\n<p>I&#8217;m continuing my experimentation and particularly with motion_bucket_id to see how it affects the animation. I will share my findings in up coming posts.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Last week&#8217;s hottest news other than the drama unfolding at Open.AI with Sam, this was a very positive news when Stability.AI announced that they have released Stable Video Diffusion (21st November 2023). A few days later the first two models are available for experimental\/evaluation use (ie. Non-commercial), so of course then we wait for the [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":10838,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"off","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[1699],"tags":[],"class_list":["post-10827","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-comfyui"],"_links":{"self":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts\/10827","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/comments?post=10827"}],"version-history":[{"count":10,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts\/10827\/revisions"}],"predecessor-version":[{"id":10856,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts\/10827\/revisions\/10856"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/media\/10838"}],"wp:attachment":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/media?parent=10827"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/categories?post=10827"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/tags?post=10827"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}