Automating Nonograms Katana: From OCR and Mouse Clicks to Save-File Reverse Engineering

I was playing Nonograms Katana - a mobile nonogram game with an embedded RPG layer where solving puzzles earns you items, coins, and progression through guild content.

The puzzles are fun. The grind is not.

Guild mode in particular expects a ridiculous amount of experience - 3,160,000 XP to reach Zen Master. At some point "one more puzzle" stops being a relaxing evening and starts feeling like a second job.

So I did what any reasonable security engineer with Python installed would do: I automated it.

Then I automated harder. Then I reverse-engineered the save file.

This is the story of that escalation - and why the final solution wasn't the solver at all.

What is a nonogram?

A nonogram (also called picross, griddlers, or paint-by-numbers) is a logic puzzle on a grid. Each row and column has a sequence of numbers - hints - that describe runs of filled cells.

For example, hints 3 1 on a row of width 10 mean: three filled cells, at least one empty gap, one filled cell, and the rest empty. You use the row and column hints together to deduce which cells must be filled and which must stay empty.

When you're done, the filled cells often form a picture - a pixel-art reward for your patience.

Small puzzles you solve with intuition. Larger ones require systematic techniques: identifying satisfied lines, finding overlap regions where a run must cover certain cells regardless of placement, and propagating constraints until the grid collapses to a unique solution - or until you need backtracking.

How computers solve them

Human solvers use pattern recognition. Programs use constraint satisfaction.

The approach I used is line solving with backtracking:

1. Treat each row and column as an independent constraint problem.
2. For a line, generate all valid placements of runs that match the hints and grid width.
3. Eliminate placements inconsistent with cells already known to be filled or empty.
4. If a line still has ambiguity, pick a cell, guess, and recurse - classic backtracking.

I didn't implement that from scratch. I used pynogram - a Python nonogram solver - with a few modifications so I could read the solved grid programmatically instead of only rendering ASCII art.

Once you have accurate hints, solving is the easy part. Getting accurate hints from a phone screen is where the pain lives.

The game I was trying to beat

Nonograms Katana wraps standard nonograms in RPG progression: guilds, items, currencies, unlock trees. You don't just solve puzzles for the picture - you solve them to fuel a meta-game loop.

That's the motivation. I didn't set out to build a computer vision pipeline because I love OCR. I set out because clicking squares for hours to earn guild resources felt like a problem that should have a programmatic answer.

Phase 1: Screenshot → OCR → Solver → Mouse

The first working design was a Windows automation loop around an Android phone mirrored with scrcpy:

[Phone: Nonograms Katana] → scrcpy mirror on Windows
         ↓
[Screenshot via pyautogui]
         ↓
[Locate nonogram frame by color]
         ↓
[Slice grid into individual hint cells]
         ↓
[OCR each cell → build pynogram input]
         ↓
[pynogram backtracking solver]
         ↓
[pyautogui moves mouse + clicks each filled cell]
         ↓
[Click Continue → Next puzzle → repeat]

Step 1: Find the puzzle on screen

image_extractor.py takes a full desktop screenshot, converts it to grayscale, and masks pixels matching the nonogram frame color (configurable RGB - in my setup, a dark green/blue border). OpenCV finds the largest square-ish contour and crops the puzzle region.

You need the mirror window large enough that hint digits remain readable. Half the screen was my minimum workable size.

Step 2: Slice the grid

image_slicer.py uses NumPy stride tricks to split the cropped image into a matrix of tiles - one image per grid cell, including the hint rows and columns around the playable area. Each tile lands in a sliced_data/ folder as img{row}_{col}.jpg.

The slicer upscales tiles (200%) before cutting because OCR on tiny digits is miserable at native resolution.

Step 3: Read the hint numbers

image_analyzer.py is the fragile heart of the pipeline. For each hint cell it:

• Converts to grayscale and applies multiple threshold variants (fixed thresholds, Otsu, Gaussian blur + Otsu, adaptive thresholds).
• Runs Tesseract OCR in parallel threads - one per threshold - and votes on the most common digit result.
• Optionally runs EasyOCR as a second opinion.
• Hashes recognized cells with OpenCV's block mean hash and caches results in a hashtable so identical hint glyphs skip OCR on future puzzles.

Successful reads get written to input.txt in pynogram's format:

[clues]
columns=3 1,0,2 1,...
rows=2,5,1 3,...

Empty hint cells become 0.

Step 4: Solve

puzzle_solver.py feeds that file to pynogram's backtracking solver and writes a binary grid to output.txt - 1 for filled, 0 for empty.

If OCR produced inconsistent hints, the solver throws. The main script retries with "hard mode" analysis (more threshold strategies and stricter voting). If that still fails, it pauses and lets you hand-edit input.txt before continuing.

Step 5: Click the answer back into the phone

mouse_event_runner.py reads the solution matrix, interpolates click coordinates between the first and last grid cell (computed during extraction), and uses pyautogui to move the cursor and click every filled square.

It looks ridiculous. It works.

Step 6: Run forever

nonogram_solver.py mode 2 - the automatic solver - wraps everything in a loop:

1. Click the first puzzle in the list.
2. Move the cursor off-screen for a clean screenshot.
3. Run the full solve pipeline.
4. Click Continue.
5. Navigate to the next puzzle.
6. Repeat.

Non-stop. Ctrl+C to stop.

There's a GIF of this - cursor moving on its own, filling squares while the game watches in silence.

Where the OCR pipeline breaks

This worked well enough for small and medium puzzles - think 10×10, 15×15 - when the mirror resolution was high and hint digits were large.

It fell apart on large puzzles:

• More hint rows and columns → more OCR calls per puzzle.
• Each hint cell shrinks on screen → Tesseract confuses 1 and 7, merges digits, returns noise.
• A single misread hint makes pynogram fail or produce a wrong grid - which the game rejects or fills incorrectly.
• Upscaling helps until it doesn't. At 80×80, my monitor simply couldn't show enough detail through scrcpy.

Higher resolution helps, but very large nonograms need something smarter than generic OCR - and I didn't have it yet.

Phase 2: Training a digit recognizer

While the OCR pipeline ran, it also harvested training data. Every analyzed hint cell saved its tile image into a temp/{digit}/ folder, building a labeled dataset under numbers/ organized by recognized value.

The next step was obvious: train a model that recognizes hint digits from images directly, tuned to Nonograms Katana's font and color scheme instead of generic Tesseract behavior.

The digit_recognizer/ subproject does exactly that:

• model_builder.py walks the numbers/ image folders.
• Resizes each glyph to 28×28, extracts FFT magnitude features from the thresholded image.
• Trains an MLPClassifier (scikit-learn) and serializes it to digit_recog_model.skops.

On a held-out test split the model reported high accuracy. In practice on live puzzle tiles - especially small ones from large grids - results were inconsistent. The integration hook in image_analyzer.py exists but stayed commented out while I iterated on features and training data.

That effort still mattered. I learned where OCR failed (ambiguous thresholds, merged digits, empty cells that aren't quite empty), collected a domain-specific dataset without labeling by hand, and validated that game automation problems eventually become ML problems - even when the ML part is harder than the puzzle solver underneath.

For large puzzles, the bottleneck was never pynogram. It was reading the hints fast and correctly.

Phase 3: Too slow - go deeper

After several days of running the automatic solver, the pattern was clear:

• Small puzzles: fast enough.
• Medium puzzles: acceptable with occasional manual input.txt fixes.
• Large puzzles: unreliable OCR, frequent retries, manual intervention.
• Net throughput: still too slow for the guild grind I was targeting.

I was optimizing the wrong layer. The game didn't care how the grid got filled. It cared about save state - XP, items, guild progress stored on disk.

So I stopped attacking the puzzle UI and started attacking the save file.

Phase 4: Reverse-engineering the save data

Nonograms Katana stores guild progress in a file called guild.dat. I pulled a save export from the game, extracted the APK, and traced how the Android client reads and writes that blob.

I wrote Java utilities to handle the format:

Decryption (RawDataGenerator.java)

The save file isn't plain JSON. Rough structure:

1. A fixed header - twelve 32-bit integers (raw_header.txt when extracted).
2. A gzip-compressed payload containing the actual guild object data.
3. A custom XOR-based cipher applied in 4-byte blocks with a rolling key (kindOfDecryption / kindOfEncryption - same routine, inverse operations).
4. A 4-byte checksum appended at the end, validated before the game accepts the file.

RawDataGenerator reads guild.dat, decrypts it, decompresses the gzip stream, and dumps the raw object bytes to raw.txt - printing hex values as it goes so you can map item quantities to inventory slots.

Editing (raw.txt)

Guild objects appear as structured byte sequences in the decrypted stream. Quantities and item IDs follow repeating patterns - once you've seen your inventory in-game, you can locate the matching rows in hex and recognize which values correspond to which items. Open the dump in a hex editor, match quantity fields against what you know you own, and edit values directly - including above normal game limits, if you're so inclined.

Re-encryption (DataMerger.java)

After editing raw.txt, DataMerger reverses the pipeline:

1. Write header from raw_header.txt.
2. Gzip-compress modified payload.
3. Apply kindOfEncryption.
4. Append fresh checksum.
5. Output guild2.dat → rename to guild.dat, bundle with NonogramsKatana.progress, re-import into the game.

That was the unlock. I could get the guild progression I wanted without solving thousands of large puzzles.

The OCR pipeline, the mouse automation, the ML experiments - all of it was an elaborate route to a destination that lived in a 2.8 KB encrypted file the whole time.

It's not production software. It's a progression of increasingly direct answers to the same question: how do I get past this grind?

Lessons (the non-toy-project kind)

1. Automate the interface first, question the interface last.
Screen scraping and mouse clicking are fragile but fast to prototype. They teach you the domain. They're rarely the final answer.

2. OCR accuracy is a resolution problem until it's a domain problem.
Generic Tesseract on game UI fonts breaks at scale. Custom ML helps - but only if your training distribution matches production tiles. I collected data organically from the pipeline itself, which is the right idea even when the model needs more iteration.

3. The real state machine is on disk.
Mobile games with RPG layers almost always persist progress in serialized, optionally encrypted files. If your goal is progression - not puzzle appreciation - that's the system to understand.

4. Reverse engineering is just threat modeling in reverse.
Find the blob. Find the checksum. Find where bytes become meaning. The same instinct that makes you ask "what happens if user input becomes computation?" makes you ask "what happens if I change these two bytes and re-sign the file?"

5. Know when to stop escalating.
I got what I wanted. The automatic solver still runs for small puzzles if I'm feeling nostalgic. The save editor is the tool I'd reach for if I ever cared about guild inventory again.

Closing

Nonograms Katana wanted me to solve puzzles. I wanted the RPG rewards without turning puzzle-solving into a full-time mouse-clicking job.

So I built an OCR pipeline, wired it to a constraint solver, taught a script to click squares through a phone mirror, trained a digit classifier on harvested screenshots, and - when all of that was still too slow - opened the save file and edited the world directly.

Was it overkill? Absolutely.

Was it more educational than grinding to Zen Master manually? Also absolutely.

If you're playing the game for fun - solve the puzzles yourself. They're better that way.

I'm back to building things that aren't nonograms. Mostly.