Add Black Forest Labs content moderation

#64
Black Forest Labs org
•
edited 3 days ago

Why

This demo generates images from any prompt and uploaded image without a content filter. This PR adds Black Forest Labs' moderation to every generation.

What changes

  • At startup, the Space downloads the shared client bfl_moderation.py from black-forest-labs/hf-spaces-moderation-client.
  • Inside the @spaces.GPU function, the prompt and any input images are checked while the image generates, and the finished image is checked before it is returned. A blocked request shows "This request was blocked by our content filter." and returns no image.
  • If the input check cannot run, the request continues, since the output check still runs. If the output check cannot run, the request fails with "The content filter is unavailable, please try again later."
  • The checks run inside the GPU function, so they count against the caller's ZeroGPU quota: infer and infer_example get duration=70.
  • Pins sentencepiece==0.2.2. The Space builder's base image pins protobuf<4, and sentencepiece 0.2.3 imports google.protobuf.runtime_version, which needs a newer protobuf, so any rebuild crashed at startup with ImportError: cannot import name 'runtime_version' from 'google.protobuf'.
  • Pins diffusers==0.41.0 instead of installing from git main. diffusers main (and 0.42.1 onward) requires Python 3.11, and this Space builds on Python 3.10, so a rebuild without a cached install failed with Package 'diffusers' requires a different Python: 3.10.13 not in '>=3.11.0'. 0.41.0 is the newest release for Python 3.10 and includes this Space's pipeline.

The API key comes from the BFL_MODERATION_KEY Space secret, which is already set on this Space.

Testing

On a private copy of this Space: benign text-to-image and benign edit → image (18.0 s, 32.0 s); explicit prompt and gory edit → blocked.

alexanderwangus changed pull request status to merged

Sign up or log in to comment