WordWeight
WordWeight lets you generate stunning images from text using cutting-edge diffusion models — then pulls back the curtain to reveal which words carried the most weight.
Overview
WordWeight takes your input prompt, and first generates an image for you. Then, when you ask for an explanation of a part of the image, wordweight computes the scalar of the sum of the products of the image mask and the actual generated image. A backward propagation follows, populating the gradients of the token embeddings. Using a vanilla gradient embedding method, each token's weight is computed and shown on the UI.
Technologies Used
Replicate
PyTorch
Azure
PostgresQL
Key Challenges
- Gradient computations beyond 8 repitions of the diffusion model were difficult because the GPU kept running out of memory. Used PyTorch checkpointing to work around this.
- Running a diffusion model in a deployed application needed a GPU, with reasonable cost. Used Replicate (an on-demand GPU allocation service) to setup the image explanation scripts.
- Needed a low cost method to deploy the application for long periods of time. Used Azure App Services, which relies on cold start mechanisms, incurring no charge to host the application online.