Duplicate from TTPlanet/TTPLanet_SDXL_Controlnet_Tile_Realistic

Co-authored-by: Aaron Xie <TTPlanet@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,116 @@
+---
+library_name: diffusers
+pipeline_tag: image-feature-extraction
+tags:
+- Controlnet
+- Tile
+- stable diffustion
+license: openrail
+---
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+Controlnet SDXL Tile model realistic version, fit for both webui extention and comfyui controlnet node.
+
+### Model Description
+
+Here's a refined version of the update notes for the Tile V2:
+
+-Introducing the new Tile V2, enhanced with a vastly improved training dataset and more extensive training steps.
+
+-The Tile V2 now automatically recognizes a wider range of objects without needing explicit prompts.
+
+-I've made significant improvements to the color offset issue. if you are still seeing the significant offset, it's normal, just adding the prompt or use a color fix node.
+
+-The control strength is more robust, allowing it to replace canny+openpose in some conditions.
+
+If you encounter the edge halo issue with t2i or i2i, particularly with i2i, ensure that the preprocessing provides the controlnet image with sufficient blurring. If the output is too sharp, it may result in a 'halo'—a pronounced shape around the edges with high contrast. In such cases, apply some blur before sending it to the controlnet. If the output is too blurry, this could be due to excessive blurring during preprocessing, or the original picture may be too small.
+
+Enjoy the enhanced capabilities of Tile V2!
+
+![TBT9$5UL`53RKP`85JXIZ_H.jpg](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/yS1ax7FWZS7b5Zz1co8_b.jpeg)
+
+![Q5A0[{{0{]I~`KJFCZJ7`}4.jpg](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/HMGmYz7IiLSqfoiMgcmgU.jpeg)
+
+<!-- Provide a longer summary of what this model is. -->
+- This is a SDXL based controlnet Tile model, trained with huggingface diffusers sets, fit for Stable diffusion SDXL controlnet.
+- It is original trained for my personal realistic model project used for Ultimate upscale process to boost the picture details. with a proper workflow, it can provide a good result for high detailed, high resolution image fix.
+- As there is no SDXL Tile available from the most open source, I decide to share this one out.
+- I will share my workflow soon as I am still working on it to have better result.
+- **I am still working on the better workflow for super upscale as I showed in the example, trust me, it's all real!!! and Enjoy**
+- 
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/ddFT3326ddNOWBeoFnfZl.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/OETMPhSCVEKdyUvILMsyp.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/lznGyTnKy91AwRmSaCxTF.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/iokmuDnYy7UC47t7AoLc1.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/gjNEgVlr2I2uf9hPJiivu.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/wSZTq340GTG3ojx75HNyH.png)
+
+
+- **Developed by:** TTPlanet
+- **Model type:** Controlnet Tile
+- **Language(s) (NLP):** No language limitation
+
+
+## Uses
+- **Important: Tile model is not a upscale model!!! it enhance or change the detial of the original size image, remember this before you use it!**
+- This model will not significant change the base model style. it only adding the features to the upscaled pixel blocks....
+- --Just use a regular controlnet model in Webui by select as tile model and use tile_resample for Ultimate Upscale script.
+- --Just use load controlnet model in comfyui and apply to control net condition.
+- --if you try to use it in webui t2i, need proper prompt setup, otherwise it will significant modify the original image color. I don't know the reason, as I don't really use this function.
+- --it do perform much better with the image from the datasets. However, everything works fine for the i2i model and what is the place usually the ultimate upscale is applied!!
+- **--Please also notice this is a realistic training set, so no comic, animation application are promised.**
+- --For tile upscale, set the denoise around 0.3-0.4 to get good result.
+- --For controlnet strength, set to 0.9 will be better choice
+- --For human image fix, IPA and early stop on controlnet will provide better reslut
+- **--Pickup a good realistic base model is important!**
+![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/zPyYn2fSFmD1Q07ME0Hkg.jpeg)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/00gDy93frzcF-WH8hh1NS.png)
+- **bsides the basic function, Tile can also change the picture style based on you model, please select the preprocessor as None(not resample!!!!) you can build different style from one single picture with great control!**
+- Just enjoy  
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/RjZiSX1oBXas1y1Tjq_dW.png)
+- 
+- **additional instruction to use this tile**
+- **Part 1：update for style change application instruction（**cloth change and keep consistent pose**）:**
+
+- 1. Open a A1111 webui.
+
+- 2. select a image you want to use for controlnet tile
+
+- 3. remember the setting is like this, make 100% preprocessor is none. and control mode is My prompt is more important.
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/RfSSfKxjpxvHSUmswTfhH.png)
+
+- 4. type in the prompts in positive and negative text box, gen the image as you wish. if you want to change the cloth, type like a woman dressed in yellow T-shirt, and change the background like in a shopping mall,
+
+- 5. Hires fix is supported!!!
+
+- 6. You will get the result as below:
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/XS-Qi-FuofnPABl5hZAoi.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/641edd91eefe94aff6de024c/KYyRUJjuxg5YKs0UFYUw0.png)
+
+- **Part2： for ultimate sd upscale application**
+
+Here is the simplified workflow just for ultimate upscale, you can modify and add pre process for your image based on the real condition. In my case, I usually make a image to image with 0.1 denoise rate for the real low quality image such as 600*400 to 1200*800 before I through it into this ultimate upscale process.
+
+Please add IPA process if you need the face likes identical, please also add IPA in the raw pre process for low quality image i2i. Remember, over resolution than downscale is always the best way to boost the quality from low resolution image.
+
+https://civitai.com/models/333060/simplified-workflow-for-ultimate-sd-upscale
+
+
+## Bias, Risks, and Limitations
+
+- **Please do not use it for adult content**
+
+### Recommendations
+
+- Use comfyui to build your own Upscale process, it works fine!!!
+
+- **Special thanks to the Controlnet builder lllyasviel Lvmin Zhang (Lyumin Zhang) who bring so much fun to us, and thanks huggingface make the training set to make the training so smooth.**
+
+
+## Model Card Contact
+
+--contact me if you want, discord with "ttplanet", Civitai with "ttplanet"
+--you can also join the group discussion with QQ gourp number: 294060503

TTPLANET_Controlnet_Tile_realistic_v1_fp16.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa2a5f2fec814819c835403c778be2c2437d147931921304569fccec2b45173c
+size 2502139104

TTPLANET_Controlnet_Tile_realistic_v1_fp32.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f33012cbac5839885f143aa2accb2e403414c7dad119364bbe18de37e4a65f67
+size 5004167864

TTPLANET_Controlnet_Tile_realistic_v2_fp16.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d7db14f6b82c981f506be138774f6d9324dd2087a193e79682b4856ec90a622
+size 2502139104

TTPLANET_Controlnet_Tile_realistic_v2_rank256.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:093d5b1c50055e57f85ee7dbf0104e6e8ae7f9dd589a6502c413146b516c6fcb
+size 774423024

TTP_tile_preprocessor_v5.py

@@ -0,0 +1,191 @@
+import cv2
+import numpy as np
+from PIL import Image
+import torch
+
+def pil2tensor(image: Image) -> torch.Tensor:
+    return torch.from_numpy(np.array(image).astype(np.float32) / 255.0).unsqueeze(0)
+
+def tensor2pil(t_image: torch.Tensor) -> Image:
+    return Image.fromarray(np.clip(255.0 * t_image.cpu().numpy().squeeze(), 0, 255).astype(np.uint8))
+
+def apply_gaussian_blur(image_np, ksize=5, sigmaX=1.0):
+    if ksize % 2 == 0:
+        ksize += 1  # ksize must be odd
+    blurred_image = cv2.GaussianBlur(image_np, (ksize, ksize), sigmaX=sigmaX)
+    return blurred_image
+
+def apply_guided_filter(image_np, radius, eps):
+    # Convert image to float32 for the guided filter
+    image_np_float = np.float32(image_np) / 255.0
+    # Apply the guided filter
+    filtered_image = cv2.ximgproc.guidedFilter(image_np_float, image_np_float, radius, eps)
+    # Scale back to uint8
+    filtered_image = np.clip(filtered_image * 255, 0, 255).astype(np.uint8)
+    return filtered_image
+
+class TTPlanet_Tile_Preprocessor_GF:
+    def __init__(self, blur_strength=3.0, radius=7, eps=0.01):
+        self.blur_strength = blur_strength
+        self.radius = radius
+        self.eps = eps
+
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "image": ("IMAGE",),
+                "scale_factor": ("FLOAT", {"default": 1.00, "min": 1.00, "max": 8.00, "step": 0.05}),
+                "blur_strength": ("FLOAT", {"default": 2.0, "min": 1.0, "max": 10.0, "step": 0.1}),
+                "radius": ("INT", {"default": 7, "min": 1, "max": 20, "step": 1}),
+                "eps": ("FLOAT", {"default": 0.01, "min": 0.001, "max": 0.1, "step": 0.001}),
+            },
+            "optional": {}
+        }
+
+    RETURN_TYPES = ("IMAGE",)
+    RETURN_NAMES = ("image_output",)
+    FUNCTION = 'process_image'
+    CATEGORY = 'TTP_TILE'
+
+    def process_image(self, image, scale_factor, blur_strength, radius, eps):
+        ret_images = []
+    
+        for i in image:
+            # Convert tensor to PIL for processing
+            _canvas = tensor2pil(torch.unsqueeze(i, 0)).convert('RGB')
+            img_np = np.array(_canvas)[:, :, ::-1]  # RGB to BGR
+            
+            # Apply Gaussian blur
+            img_np = apply_gaussian_blur(img_np, ksize=int(blur_strength), sigmaX=blur_strength / 2)            
+
+            # Apply Guided Filter
+            img_np = apply_guided_filter(img_np, radius, eps)
+
+
+            # Resize image
+            height, width = img_np.shape[:2]
+            new_width = int(width / scale_factor)
+            new_height = int(height / scale_factor)
+            resized_down = cv2.resize(img_np, (new_width, new_height), interpolation=cv2.INTER_AREA)
+            resized_img = cv2.resize(resized_down, (width, height), interpolation=cv2.INTER_CUBIC)
+            
+
+
+            # Convert OpenCV back to PIL and then to tensor
+            pil_img = Image.fromarray(resized_img[:, :, ::-1])  # BGR to RGB
+            tensor_img = pil2tensor(pil_img)
+            ret_images.append(tensor_img)
+        
+        return (torch.cat(ret_images, dim=0),)
+        
+class TTPlanet_Tile_Preprocessor_Simple:
+    def __init__(self, blur_strength=3.0):
+        self.blur_strength = blur_strength
+
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "image": ("IMAGE",),
+                "scale_factor": ("FLOAT", {"default": 2.00, "min": 1.00, "max": 8.00, "step": 0.05}),
+                "blur_strength": ("FLOAT", {"default": 1.0, "min": 1.0, "max": 20.0, "step": 0.1}),
+            },
+            "optional": {}
+        }
+
+    RETURN_TYPES = ("IMAGE",)
+    RETURN_NAMES = ("image_output",)
+    FUNCTION = 'process_image'
+    CATEGORY = 'TTP_TILE'
+
+    def process_image(self, image, scale_factor, blur_strength):
+        ret_images = []
+    
+        for i in image:
+            # Convert tensor to PIL for processing
+            _canvas = tensor2pil(torch.unsqueeze(i, 0)).convert('RGB')
+        
+            # Convert PIL image to OpenCV format
+            img_np = np.array(_canvas)[:, :, ::-1]  # RGB to BGR
+        
+            # Resize image first if you want blur to apply after resizing
+            height, width = img_np.shape[:2]
+            new_width = int(width / scale_factor)
+            new_height = int(height / scale_factor)
+            resized_down = cv2.resize(img_np, (new_width, new_height), interpolation=cv2.INTER_AREA)
+            resized_img = cv2.resize(resized_down, (width, height), interpolation=cv2.INTER_LANCZOS4)
+        
+            # Apply Gaussian blur after resizing
+            img_np = apply_gaussian_blur(resized_img, ksize=int(blur_strength), sigmaX=blur_strength / 2)
+        
+            # Convert OpenCV back to PIL and then to tensor
+            _canvas = Image.fromarray(img_np[:, :, ::-1])  # BGR to RGB
+            tensor_img = pil2tensor(_canvas)
+            ret_images.append(tensor_img)
+    
+        return (torch.cat(ret_images, dim=0),)        
+
+class TTPlanet_Tile_Preprocessor_cufoff:
+    def __init__(self, blur_strength=3.0, cutoff_frequency=30, filter_strength=1.0):
+        self.blur_strength = blur_strength
+        self.cutoff_frequency = cutoff_frequency
+        self.filter_strength = filter_strength
+
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "image": ("IMAGE",),
+                "scale_factor": ("FLOAT", {"default": 1.00, "min": 1.00, "max": 8.00, "step": 0.05}),
+                "blur_strength": ("FLOAT", {"default": 2.0, "min": 1.0, "max": 10.0, "step": 0.1}),
+                "cutoff_frequency": ("INT", {"default": 100, "min": 0, "max": 256, "step": 1}),
+                "filter_strength": ("FLOAT", {"default": 1.0, "min": 0.1, "max": 10.0, "step": 0.1}),
+            },
+            "optional": {}
+        }
+
+    RETURN_TYPES = ("IMAGE",)
+    RETURN_NAMES = ("image_output",)
+    FUNCTION = 'process_image'
+    CATEGORY = 'TTP_TILE'
+
+    def process_image(self, image, scale_factor, blur_strength, cutoff_frequency, filter_strength):
+        ret_images = []
+    
+        for i in image:
+            # Convert tensor to PIL for processing
+            _canvas = tensor2pil(torch.unsqueeze(i, 0)).convert('RGB')
+            img_np = np.array(_canvas)[:, :, ::-1]  # RGB to BGR
+
+            # Apply low pass filter with new strength parameter
+            img_np = apply_low_pass_filter(img_np, cutoff_frequency, filter_strength)
+
+            # Resize image
+            height, width = img_np.shape[:2]
+            new_width = int(width / scale_factor)
+            new_height = int(height / scale_factor)
+            resized_down = cv2.resize(img_np, (new_width, new_height), interpolation=cv2.INTER_AREA)
+            resized_img = cv2.resize(resized_down, (width, height), interpolation=cv2.INTER_LANCZOS4)
+            
+            # Apply Gaussian blur
+            img_np = apply_gaussian_blur(img_np, ksize=int(blur_strength), sigmaX=blur_strength / 2)
+            
+            # Convert OpenCV back to PIL and then to tensor
+            pil_img = Image.fromarray(resized_img[:, :, ::-1])  # BGR to RGB
+            tensor_img = pil2tensor(pil_img)
+            ret_images.append(tensor_img)
+        
+        return (torch.cat(ret_images, dim=0),)
+
+NODE_CLASS_MAPPINGS = {
+    "TTPlanet_Tile_Preprocessor_GF": TTPlanet_Tile_Preprocessor_GF,
+    "TTPlanet_Tile_Preprocessor_Simple": TTPlanet_Tile_Preprocessor_Simple,
+    "TTPlanet_Tile_Preprocessor_cufoff": TTPlanet_Tile_Preprocessor_cufoff
+}
+
+NODE_DISPLAY_NAME_MAPPINGS = {
+    "TTPlanet_Tile_Preprocessor_GF": "🪐TTPlanet Tile Preprocessor GF",
+    "TTPlanet_Tile_Preprocessor_Simple": "🪐TTPlanet Tile Preprocessor Simple",
+    "TTPlanet_Tile_Preprocessor_cufoff": "🪐TTPlanet Tile Preprocessor cufoff"
+}

config.json

@@ -0,0 +1,57 @@
+{
+  "_class_name": "ControlNetModel",
+  "_diffusers_version": "0.27.0.dev0",
+  "act_fn": "silu",
+  "addition_embed_type": "text_time",
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": 256,
+  "attention_head_dim": [
+    5,
+    10,
+    20
+  ],
+  "block_out_channels": [
+    320,
+    640,
+    1280
+  ],
+  "class_embed_type": null,
+  "conditioning_channels": 3,
+  "conditioning_embedding_out_channels": [
+    16,
+    32,
+    96,
+    256
+  ],
+  "controlnet_conditioning_channel_order": "rgb",
+  "cross_attention_dim": 2048,
+  "down_block_types": [
+    "DownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "global_pool_conditions": false,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "mid_block_type": "UNetMidBlock2DCrossAttn",
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "projection_class_embeddings_input_dim": 2816,
+  "resnet_time_scale_shift": "default",
+  "transformer_layers_per_block": [
+    1,
+    2,
+    10
+  ],
+  "upcast_attention": null,
+  "use_linear_projection": true
+}