je-suis-tm/wong_kar_wai_my_blueberry_nights_lora_flux2_nf4

Wong Kar Wai My Blueberry Nights style Lora Flux2 NF4

Prompt

Training With QLoRA: WKW style, A man, dark hair, clean shaved, in a white shirt patterned with red and green florals. His eyes are closed as if he is meditating. The background is indistinct, a wash of muted tones. Two figures are faintly visible further back, their details lost in the blur. The overall impression is one of quiet contemplation and obscured context.

Prompt

Training With QLoRA: WKW style, A woman, dark hair cascading, stands behind a bar, cradling a red drink. She wears a short sleeved black leopard print dress, a flash of gold light illuminates her left arm. Her right hand holds a glass of whiskey high up in the air as if she is proposing a toast. her face is obscured by the darkness. Leaning on the bar, she exudes a sad vibe. Two dark leather barstools sit nearby. The blurred background hints at a dimly lit, retro space, with leopard print details. Warm lighting and classic decor create a nostalgic, sophisticated atmosphere, suggesting a moment of melancholy.

Prompt

Training With QLoRA: WKW style, A woman in a black lace dress stands before a jukebox, bathed in the crimson glow of nighttime industrial lights. Facing away from the camera, her dark, wavy hair frames a contemplative pose. The blurry shadowy background reveals a bar. The scene evokes a palpable sense of mystery and delicate femininity.

Prompt

Training With QLoRA: WKW style, a woman, dark hair cascading, head thrown back, mouth agape, stunned disbelief. green lights illuminate her face. Stark black wall backdrop. Dominating the scene, a vibrant green neon sign. The overall impression is one of unsettling drama and isolation, a moment frozen in a harsh, artificial light.

Prompt

Training With QLoRA: WKW style, A tender connection lingers. A man, green beanie perched jauntily, is met with a woman with both of them smiling with closed eyes. The scene, bathed in the warm glow of a storefront window, partially veiled by the words, giving the vibe of happiness. It's a fleeting moment, likely plucked from a romantic drama, radiating a quiet nostalgia. A simple, beautiful exchange.

Prompt

Training With QLoRA: WKW style, A man, sporting short brown hair, sits precariously on a bar counter during mid-food prep. He reaches for a plate with food. The background is under a soft blue hue. Wooden chairs and small tables with paper placemats populate the background.

Prompt

Training Without QLoRA: WKW style, A man, dark hair, clean shaved, in a white shirt patterned with red and green florals. His eyes are closed as if he is meditating. The background is indistinct, a wash of muted tones. Two figures are faintly visible further back, their details lost in the blur. The overall impression is one of quiet contemplation and obscured context.

Prompt

Training Without QLoRA: WKW style, A woman, dark hair cascading, stands behind a bar, cradling a red drink. She wears a short sleeved black leopard print dress, a flash of gold light illuminates her left arm. Her right hand holds a glass of whiskey high up in the air as if she is proposing a toast. her face is obscured by the darkness. Leaning on the bar, she exudes a sad vibe. Two dark leather barstools sit nearby. The blurred background hints at a dimly lit, retro space, with leopard print details. Warm lighting and classic decor create a nostalgic, sophisticated atmosphere, suggesting a moment of melancholy.

Prompt

Training Without QLoRA: WKW style, A woman in a black lace dress stands before a jukebox, bathed in the crimson glow of nighttime industrial lights. Facing away from the camera, her dark, wavy hair frames a contemplative pose. The blurry shadowy background reveals a bar. The scene evokes a palpable sense of mystery and delicate femininity.

Prompt

Training Without QLoRA: WKW style, a woman, dark hair cascading, head thrown back, mouth agape, stunned disbelief. green lights illuminate her face. Stark black wall backdrop. Dominating the scene, a vibrant green neon sign. The overall impression is one of unsettling drama and isolation, a moment frozen in a harsh, artificial light.

Prompt

Training Without QLoRA: WKW style, A tender connection lingers. A man, green beanie perched jauntily, is met with a woman with both of them smiling with closed eyes. The scene, bathed in the warm glow of a storefront window, partially veiled by the words, giving the vibe of happiness. It's a fleeting moment, likely plucked from a romantic drama, radiating a quiet nostalgia. A simple, beautiful exchange.

Prompt

Training Without QLoRA: WKW style, A man, sporting short brown hair, sits precariously on a bar counter during mid-food prep. He reaches for a plate with food. The background is under a soft blue hue. Wooden chairs and small tables with paper placemats populate the background.

Prompt

Testing With QLoRA: WKW Style, A captivating portrait depicts a woman with short, dark hair, her gaze fixed to her right. Centrally framed, she wears a black lace top or dress. Her head is subtly angled towards the viewer. The background is softly blurred, hinting at a red wall with vertical stripes and a vibrant, patterned corner. The overall mood is contemplative and evocative.

Prompt

Testing With QLoRA: WKW Style, A woman in a yellow dress with a lace trimmed cowboy hat poses against a weathered brick column. tan brick columns forms the backdrop, punctuated by a glimpse of a vehicle.

Prompt

Testing With QLoRA: WKW Style, A woman stands on a city street at night, facing left. Her dark hair is in an updo, and she wears a vibrant red top with white stitching. Blurred lights in the background hint at a busy urban scene, streetlights and building windows glow indistinctly. The woman's neutral expression creates a sense of mystery, leaving her thoughts and the scene's narrative open to interpretation.

Prompt

Testing Without QLoRA: WKW Style, A captivating portrait depicts a woman with short, dark hair, her gaze fixed to her right. Centrally framed, she wears a black lace top or dress. Her head is subtly angled towards the viewer. The background is softly blurred, hinting at a red wall with vertical stripes and a vibrant, patterned corner. The overall mood is contemplative and evocative.

Prompt

Testing Without QLoRA: WKW Style, A woman in a yellow dress with a lace trimmed cowboy hat poses against a weathered brick column. tan brick columns forms the backdrop, punctuated by a glimpse of a vehicle.

Prompt

Testing Without QLoRA: WKW Style, A woman stands on a city street at night, facing left. Her dark hair is in an updo, and she wears a vibrant red top with white stitching. Blurred lights in the background hint at a busy urban scene, streetlights and building windows glow indistinctly. The woman's neutral expression creates a sense of mystery, leaving her thoughts and the scene's narrative open to interpretation.

All files are also archived in https://github.com/je-suis-tm/huggingface-archive in case this gets censored.

Wong Kar Wai is one of my fav directors. This QLoRA intends to replicate that grainy, saturated jewel tone, dimly-lit vibe of the movie My Blueberry Nights (2007).

The training is based on https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/test_dreambooth_lora_flux2.py. Everything in this training script needs to be set at the same torch dtype as the script was designed for unquantized version which will export in float32. The training took 14 hours on A100 80GB with max VRAM consumption at 32GB. The inference consumes 35GB of VRAM. To avoid running low on VRAM, both transformers and text_encoder were quantized.

Train

export MODEL_NAME="diffusers/FLUX.2-dev-bnb-4bit"
export INSTANCE_DIR="/pvol/wong_kar_wai_my_blueberry_nights"
export OUTPUT_DIR="/pvol/wong_kar_wai_my_blueberry_nights_lora_flux2_nf4"
export Q_DIR="/pvol/quantization_config.json"
accelerate config default
accelerate launch train_dreambooth_lora_flux2.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --mixed_precision="bf16" \
  --dataset_name=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --bnb_quantization_config_path=$Q_DIR \
  --gradient_checkpointing \
  --cache_latents \
  --instance_prompt="WKW style" \
  --caption_column="text" \
  --aspect_ratio_buckets="800,1904" \
  --center_crop \
  --train_batch_size=1 \
  --guidance_scale=1 \
  --use_8bit_adam \
  --offload \
  --checkpointing_steps=100 \
  --gradient_accumulation_steps=4 \
  --optimizer="adamW" \
  --learning_rate=1e-4 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=100 \
  --max_train_steps=1500 \
  --rank=4 \
  --seed="0" 

Usage

import torch
from transformers import Mistral3ForConditionalGeneration

from diffusers import Flux2Pipeline, Flux2Transformer2DModel

repo_id = "diffusers/FLUX.2-dev-bnb-4bit"
device = "cuda:0"
torch_dtype = torch.float32 #only supports float32 when using train_dreambooth_lora_flux2.py 

transformer = Flux2Transformer2DModel.from_pretrained(
  repo_id, subfolder="transformer", torch_dtype=torch_dtype, device_map="cuda:0"
)
text_encoder = Mistral3ForConditionalGeneration.from_pretrained(
  repo_id, subfolder="text_encoder", dtype=torch_dtype, device_map="cuda:0"
)

pipe = Flux2Pipeline.from_pretrained(
  repo_id, transformer=transformer, text_encoder=text_encoder, torch_dtype=torch_dtype
)
pipe.load_lora_weights("je-suis-tm/wong_kar_wai_my_blueberry_nights_lora_flux2_nf4",
                       weight_name='pytorch_lora_weights.safetensors')
pipe.enable_model_cpu_offload()

prompt = "WKW Style, A woman in a yellow dress with a lace trimmed cowboy hat poses against a weathered brick column. tan brick columns forms the backdrop, punctuated by a glimpse of a vehicle"

image = pipe(
  prompt=prompt,
  generator=torch.Generator(device=device).manual_seed(42),
  num_inference_steps=50, # 28 is a good trade-off
  guidance_scale=4,
  height=800, 
  width=1904, #the movie was in a weird resolution 800*1912, the resolution needs to be divided by 16 so 1904
).images[0]

image.save("wong_kar_wai_my_blueberry_nights.png")

Trigger words

You should use WKW style to trigger the image generation.

Download model

Download them in the Files & versions tab.