Personalized Image Generation By Text-To-Image Diffusion Models

This paper investigates subject-driven image generation using text-to-image diffusion models with a primary focus on disentanglement. Subject-driven generation aims to generate specific concepts ("my pet dog") in various contexts using diffusion models like Stable Diffusion. However, these models often suffer from the entanglement of subject identity with the background and other features, leading to artifacts and inaccuracies. Several established methods are compared, including Textual Inversion, DreamBooth, Custom Diffusion, and DisenBooth, with DisenBooth being analyzed in depth due to its unique approach to disentanglement. The paper proposes new data augmentation techniques and introduces trajectory plots as an evaluation method to visualize the trade-off between textual and visual alignment. Additionally, the paper explores several modifications to DisenBooth, including the usage of additional or modified loss functions, CLIP reference injection, and adapter-free approaches, to assess their impact on the disentanglement problem. The results reveal that while methods like DisenBooth show promise, disentanglement is still imperfect. Issues such as concept-context leakage and the lack of the generalization of identityirrelevant embeddings persist. The adapter is too weak for the method to be benefited from the augmentation. Also, changes to contrastive loss proves ineffective.