Checking a photo against a challenge

Every photo in Kapmo is checked against its challenge before it goes into the diary. If the challenge is “find a small pattern hiding in something ordinary” and you photograph your shoe, you should hear about it. The check itself is fairly simple. Writing good challenges is the harder part, and that’s a post for another time.

How the check works now

When you take a photo, the app sends it to an edge function on Supabase along with the challenge. The function gives both to a vision model and asks it to judge.

The instructions are short. The model is told to be a fair judge, and to be generous with creative interpretations as long as the attempt is clear. It should only reject a photo if the subject is unrelated, there’s nothing meaningful in it, or it’s too blurry or dark to tell. It scores the photo from 0 to 100, and anything at 50 or above passes.

It also writes two or three sentences of feedback in your language, starting with something positive and then at most one suggestion.

There’s one cheat check. If the main subject of the photo is a screen showing a picture you found somewhere else, it fails. A TV in the background or a reflection in a window is fine.

The model answers in JSON against a fixed schema, so the app always gets the same fields back: whether it passed, the score, how confident the model is, the feedback, and one sentence on why.

When the check can’t run

Two things can go wrong, and they’re handled differently.

If the model can’t be reached at all, the photo goes through. It gets a neutral score of 60 and a note that the system fell back. A problem in the backend shouldn’t stop people from using the app, and blocking every photo would break the whole loop.

If the model answers but its reply can’t be read, the photo doesn’t pass, and you’re asked to try again.

Other ways to do it

Sending the whole image to one vision model and asking whether it matches is the simplest option, and it’s what Kapmo does now. There are two others I’ve looked at.

One has two steps. A cheaper vision model describes the photo in text, and then a stronger model compares that description against the challenge more carefully. That can make the judging smarter for less money, but it takes two calls, and if the description leaves out the detail that matters, the judge never sees it.

The other is a decision model, like Jev or OpenAI’s Decisions API. Whether a photo passes is a yes or no question, which is exactly what these models are for. Instead of a score the model makes up, you get a probability that the answer is yes, and you pick the threshold yourself. Picking that threshold well means collecting some photos that should and shouldn’t pass and tuning against them.

How it will work after the redesign

The whole app is being redesigned, and I’ll write about that in its own post. The check changes a lot with it, mostly in what you see.

You take the photo in a paper-frame camera, with the challenge always shown above the viewfinder, so you don’t forget what you’re looking for. After the shutter, a short Checking screen shows while the photo is judged.

If it passes, you land on a screen that says “Saved to your diary”, with your photo, the challenge and its instruction underneath, and buttons to continue or open the diary. There’s no feedback on an accepted photo anymore. Today the model writes a few sentences about every photo, but once a photo is in, there’s nothing to improve, so the redesign drops it.

If it doesn’t pass, you get a hint, something like “Try a different angle”, with one button to try again. Feedback only exists to help you get the photo on the next go.

If you’re offline, the camera still works. Your photo is held on screen with a note that it’s kept and will be checked when you’re back online, plus buttons to retake it or check it. Nothing is passed or failed without the check. That replaces today’s rule of letting photos through when the checker can’t be reached, and it should almost never come up, since the model provider has been very stable.

Guests get one photo before they need an account. If it passes, it’s kept on the phone and added to their diary when they sign up.

All of this makes the model’s job smaller. It only needs to answer yes or no, and give a hint when the answer is no. That’s exactly the shape of a decision model, so the redesign will most likely use one: a probability for the yes or no, with a threshold I can tune, and a short hint only when a photo is rejected.