Building Emoji Studio: A Project for build-small-hackathon
Introduction
Have you ever thought 'there really should be an emoji for this'? I have. Constantly.
But here's what makes this more interesting: LLMs use emojis. They use them naturally in responses, and they're genuinely expressive when they do, allowing LLMs to express themselves better visually. The catch is that they're locked into whatever emoji vocabulary existed in their training data. They can't use something they've never seen. Furthermore, some emojis are more popular than others, which affects how likely the model is to reach for them.
So I started wondering: what if you could just... invent new ones and teach an LLM to use them when you talk to it?
The Idea
The goal of Emoji Studio is to let you design custom emojis and then actually teach an LLM to use them in conversation.
There are two ways I thought I could approach this:
Option 1 (add the emoji as a new token to the model's vocabulary). Theoretically elegant. In practice: you'd need to retrain or fine-tune the model, it's computationally expensive, and you lose the ability to swap things out easily.
Option 2 (inject the emoji into the system prompt for in-context learning). The model sees a list of emoji names and descriptions before the conversation starts, and learns on the fly. Simpler, surprisingly robust, and something you can iterate on in minutes rather than weeks. The tradeoff is context window cost (it doesn't scale infinitely), but for a hackathon project, it's the right call.
I went with Option 2.
High Level Plan
Just ask for one in natural language, for example, "make me a happy hippo emoji" or "create an emoji of a steaming pizza slice." The assistant will generate a brand-new image for it automatically. Once your emoji is generated, you'll be asked what it means and when you'd use it (e.g. "feeling excited about food" or "use when something is hot"). This helps the assistant know when it's appropriate to use your emoji later. If you are unhappy with the generated emoji or want to abort, just type 'cancel'. Once saved, your emoji becomes available in two ways:
- Click the emoji picker button next to the message box to browse and insert any of your custom emojis directly into your message.
- The assistant can also use your emojis on its own, weaving them naturally into its replies when it fits the context.
Every emoji you create is added to your personal collection and stays available throughout your session. Hover over an emoji in the picker to see its name. The Clear button resets the conversation but keeps your custom emoji collection intact, so you don't have to recreate them.
Internal Logic
Emoji Studio is powered by 3 models working together: Qwen3-8B handles the conversation, FLUX.1-schnell handles image generation for the emoji itself, and Rembg handles background removal. The first two are run through the Hugging Face Inference API, and the whole app is hosted as a Hugging Face Space.
Qwen reads your message along with the list of emojis you've created so far, and decides how to respond (including when to call a special tool to create a new emoji).
BASE_SYSTEM_PROMPT = """\
You are a helpful assistant in a chat app that supports custom emojis.
Custom emojis are inserted with <emoji>name</emoji> syntax. When you use a \
registered emoji name inside those tags it renders as an image.
RULES:
- DEFAULT: use NO custom emoji. This is the default for every message, including greetings, \
small talk, and generic replies.
- Do NOT use standard Unicode emoji characters at the same time as custom emojis <emoji>name</emoji>.
- STRICT MATCH ONLY: insert a custom emoji ONLY if the user's message explicitly expresses or describes \
the exact situation in that emoji's "meaning" (e.g. the user states they feel sad, stressed, down, etc. \
for a "cheer up when sad" emoji). A neutral or generic message (e.g. "hi", "hello", "how are you", \
"ok", "thanks") NEVER qualifies, even if the emoji's meaning is broadly positive or cheerful.
- If you are unsure whether a message qualifies, do NOT use the emoji.
- Only use names from the "Registered emojis" list. Never guess or invent names.
- Never responds with a single emoji alone without additional text.
- Never refer to or describe an emoji in text (e.g. "here's a", "have this emoji"). \
Insert emoji only as a decorative inline addition to add meaning.
- When the user asks for a new emoji/sticker/icon, call the generate_emoji tool \
with a fitting name and description. Do not ask the user for the name or description.
- After the tool call your job is done — the system handles the follow-up flow.
(No custom emojis registered yet.)
OR
Registered emojis:
- name: "happy_hippo" | meaning: Use this emoji when I'm feeling sad to cheer me up.
- ...
"""
The "generate_emoji" tool
To let Qwen actually create something rather than just talk about it, it's given one tool: generate_emoji. The schema is intentionally minimal: just a short snake_case name for the emoji and a visual description that gets handed off to the image model.
GENERATE_EMOJI_TOOL = {
"type": "function",
"function": {
"name": "generate_emoji",
"description": (
"Generates a brand-new emoji-style image. "
"Call this when the user asks for a new emoji, sticker, or icon. "
"Provide a short unique snake_case name and a visual description for image generation."
),
"parameters": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Short unique snake_case identifier, e.g. 'pizza_slice' or 'happy_cat'",
},
"description": {
"type": "string",
"description": "Visual description used to generate the image, e.g. 'a steaming pizza slice with melted cheese'",
},
},
"required": ["name", "description"],
},
},
}
So when you type something like "make me a steaming pizza slice emoji," Qwen doesn't try to describe an emoji in words. Instead it fires off a tool call that looks roughly like this:
<tool_call>
{
"name": "generate_emoji",
"arguments": {
"name": "pizza_slice",
"description": "a steaming pizza slice with melted cheese"
}
}
</tool_call>
The app intercepts that tool call, extracts the name and description, and hands the description straight to the image generation step. From Qwen's perspective, its job ends the moment the tool call is made; the rest of the flow (image generation, background removal, asking the user what the emoji means) is handled by the app itself.
Generating the emoji image
Once we have a description, FLUX.1-schnell turns it into an actual image. The trick here is prompt engineering: FLUX doesn't know it's making an "emoji" unless you tell it very specifically what that means (flat, centered, cropped tightly, no background clutter). Here's the function that wraps all of that:
def generate_emoji_image(description: str) -> Image.Image:
prompt = (
"Apple iPhone emoji style, clean flat vector emoji illustration, "
"single subject only, large subject filling 80-90% of the frame, "
"tight crop, minimal padding, centered composition, "
"no extra background elements, no text, no border, "
"white background, high contrast, emoji-sized readability: "
f"{description}"
)
image = client.text_to_image(
prompt=prompt,
model=FLUX_MODEL,
width=512,
height=512,
)
image = image.convert("RGBA")
image = remove(image)
return image
A few things worth calling out:
- The user's description (e.g. "a steaming pizza slice with melted cheese") is wrapped in a fixed style prefix. This keeps every emoji visually consistent (flat, centered, high-contrast) no matter what subject is requested.
- The white background isn't a stylistic choice so much as a practical one: it gives
rembga clean, uniform surface to strip away. - After generation,
remove()(from therembglibrary) converts the white background into transparency, so the final PNG looks like a proper sticker rather than a square image with a white box around it.
From generation to your collection
Once FLUX has produced the image and the background has been removed, the app asks: "What should this emoji mean, and when would you use it?" Your reply is stored as-is as the emoji's meaning, and the whole thing (name, image, and meaning) gets appended to your emoji inventory for the session.
From that point on, two things happen automatically:
- The emoji shows up in the picker next to the message box, ready for you to insert manually.
- Its name and meaning get added to the "Registered emojis" section of the system prompt, so Qwen knows it exists and (per the strict-match rule) only reaches for it when your message genuinely matches that meaning.
The way the assistant actually inserts an emoji is through the same <emoji>name</emoji> tag syntax described in the system prompt. On the way out, the app scans the reply for those tags and swaps each one for the corresponding base64-encoded image, so what you see rendered in the chat is the actual emoji sticker, not raw text.
As a small safety net, if Qwen ever produces a tag for a name that isn't in your inventory (despite the system prompt telling it not to), that tag is silently dropped rather than leaked into the chat as raw <emoji>...</emoji> text, so a stray hallucinated tag just disappears instead of showing up as garbled markup.
Finally, Gradio ties the whole thing together as the frontend: the chat window, the emoji picker, the typing indicator, and the info overlay are all built on top of it, with a bit of custom CSS and JS layered in for the picker and composer behavior.
Wrapping up
What I like most about this isn't the engineering, it's what it unlocks. Every emoji you've ever used was designed by someone else, for a generic audience. Here, if you want an emoji for a very specific feeling, mood, or in-joke that no existing emoji quite captures, you just describe it and it exists. The model picks up your custom vocabulary and starts weaving it into replies, so over a conversation you build up a little shared language, including ways to express emotions that standard emoji sets just don't cover.
The catch is context size: every emoji adds a line to the system prompt, re-sent with every message. Fine for a handful, but it doesn't scale to small models and a huge collection without smarter retrieval or eventually baking popular ones into the model itself.
If you want to try it yourself or look at the code, it was built for the Hugging Face Build Small Hackathon. Give it a try and see what emojis you come up with. 🤗