The Wayback Machine - https://web.archive.org/web/20230204083640/https://github.com/huggingface/transformers/pull/21329
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add VQGAN-CLIP research project #21329

Merged
merged 17 commits into from Feb 2, 2023
Merged

Conversation

ErwannMillon
Copy link
Contributor

What does this PR do?

Implements VQGAN-CLIP using huggingface CLIP models

Related to #21064

This Research Project allows users to generate or edit images with a single line of code. It wraps the huggingface CLIPProcessor class, allowing images to be processed as torch tensors in order to preserve gradient flow through the transformations.

Features:

  • Positive and negative prompts
  • Multiple prompts
  • Prompt Weights
  • Creating GIF animations of the transformations
  • Wandb logging

Tagging @amyeroberts for review :)

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Jan 26, 2023

The documentation is not available anymore as the PR was closed or merged.

Copy link
Contributor

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice - looking good! Thanks for adding this example.

I've made a few comments, mostly nits. None are critical and I'll leave it to you to decide what you do or don't want to change.

One thing to update is the face.jpeg image. To avoid adding lots of data to the repo, could you do the same as in other examples, and upload the image to a dataset where it can be display through hrefs and downloaded from e.g.

<img src="https://huggingface.co/datasets/lvwerra/repo-images/raw/main/code-highlighting-streamlit.png" alt="drawing" width="350"/>
?

examples/research_projects/vqgan-clip/VQGAN_CLIP.py Outdated Show resolved Hide resolved
examples/research_projects/vqgan-clip/requirements.txt Outdated Show resolved Hide resolved
examples/research_projects/vqgan-clip/README.md Outdated Show resolved Hide resolved
examples/research_projects/vqgan-clip/VQGAN_CLIP.py Outdated Show resolved Hide resolved
examples/research_projects/vqgan-clip/img_processing.py Outdated Show resolved Hide resolved
examples/research_projects/vqgan-clip/VQGAN_CLIP.py Outdated Show resolved Hide resolved
ErwannMillon and others added 11 commits January 30, 2023 14:58
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
@ErwannMillon
Copy link
Contributor Author

Hi there,
Thanks for the feedback, I fixed the style issues and removed the face image.
Also refactored the code to have more accurate function names, used just the tokenizer, changed the assertions to exceptions, and removed some extraneous code (eg double crop, freeze_module)
Have a great day :)
Erwann

@amyeroberts
Copy link
Contributor

@ErwannMillon - thanks for the updates. It's looking good 😎 !

Just two last things before I think we're ready to merge:

  • Removing the face.jpg file and instead pointing to a place where it can be downloaded
  • Could you also remove the notebook? Like the image, we want to avoid adding large files as much as possible. Happy for you to link to e.g. a colab which shows this demo in the README.
Copy link
Contributor

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to merge from my end :)

examples/research_projects/vqgan-clip/VQGAN_CLIP.py Outdated Show resolved Hide resolved
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
sgugger
sgugger approved these changes Feb 2, 2023
Copy link
Member

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! Can we just remove the empty folder model_checkpoints from the PR?

sgugger
sgugger approved these changes Feb 2, 2023
@sgugger sgugger merged commit ea55bd8 into huggingface:main Feb 2, 2023
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
4 participants