I spent weeks building a working application. I lost around three days fighting the commas in my own team’s Excel sheets.
These figures are from memory. I did not keep detailed logs. They are close to what actually happened.
The Bottleneck Nobody Budgets For
O9X was ready for live testing. The code worked. Users were waiting.
Then I tried to move the data in.
Goldratt’s Theory of Constraints: the output of any system is set by its weakest link. Remove that link first. Everything else is irrelevant until then. In building O9X, data migration felt like that link. Clean code, working schema, live users — none of it works if the data cannot get in.
The code is what you build. The data migration is what makes it real.
In Log 3, I documented the archaeology problem — ten years of records scattered across legacy payroll, Google Drive, WhatsApp, Airtable, and Excel. The plan was to stop the bleeding first and migrate history slowly.
What I did not anticipate was how badly the transfer would go, even for data I had already cleaned.
Where the Assumption Broke
I had mirrored our payroll system and Zoho Books when building O9X. Same field names. Same taxonomy. Same logic. I assumed migration would be a simple export and import.
This was planning fallacy. I had solved the hard visible problem — matching the taxonomy — and my brain marked migration as done. The steps remaining after the visible work finished were invisible to me until they failed.
Matching taxonomies does not mean matching data.
My first migration plan used CSV files. Standard approach. Convert from Excel, import into O9X.
The first problem surfaced immediately. Bank account numbers in India often start with leading zeroes. My team had saved them as numbers in Excel. Excel dropped the zeroes on conversion. A 16-digit account number silently became 15 digits. The data looked complete. It wasn’t.
The second problem was worse. CSV uses commas as delimiters. My existing data — addresses, notes, descriptions — contained commas as part of the text. When the file converted, those commas broke the column structure. Fields shifted. Data landed in the wrong columns. Some records corrupted entirely.
Neither failure showed up as an error. The import completed. The damage was invisible until I verified individual records manually.
The Rebuild
I scrapped the CSV pipeline and rebuilt it around Excel. Excel handles leading zeroes correctly if you format the column. It handles commas in text because it does not use commas as delimiters. The problems that broke CSV do not exist in Excel.
Rebuilding the pipeline meant rebuilding validation logic, error handling, and data mapping. From scratch. Every time I thought I had covered the edge cases, more appeared — roughly 30 in total, text importing as numbers and breaking downstream logic each time.
Around three days. One complete rebuild.
My team had saved data the way Excel trained them to. Numbers without zeroes. Commas inside address fields. Once I identified the problem, I trained them to format differently. The issue was not carelessness. It was a gap between how ops teams naturally use spreadsheets and what an import pipeline expects. Once you know the gap, you correct for it. That correction is now part of our onboarding.
The Schema Threat Running Underneath
While I was fixing the import pipeline, AI was suggesting schema changes to handle the edge cases.
This is where the real risk lived.
Earlier in this build — before any live users — one schema change corrupted the database entirely. No data in the system at the time, so I deleted it, restored from a saved copy of the older schema, and rebuilt. Painful but recoverable.
With live users now in the system, that option was gone.
I stopped accepting AI schema suggestions directly. Instead I opened a separate Claude Code session. I gave it access to Supabase MCP with my project config and Context7 MCP for live documentation. Then I set one fixed mandate in the claude.md file:
“You are an expert Supabase engineer who has worked across 1,000+ projects and understand that more than big changes, it’s the small changes that cause issues that persist for long. Some errors can be fixed and some can’t, resulting in project failure or data loss. When I share a change in my database schema, analyse it against the existing schema using Supabase MCP and understand the latest documentation via Context7 MCP. Analyse the risk on a scale of one to ten and explain clearly what this change does, what risks it carries, and how to mitigate them.”
The first AI wrote the import code. This session audited every schema change before I touched anything. I explained the issue at hand and why a schema change might be needed. It explained the risk. I held the veto.
My AI session flagged two changes as high risk. Neither was obvious from the code alone.
The first was a Row Level Security change that would have given a class of users edit access. That class should only ever create and read — never edit. The AI had proposed the change to simplify a query. The simplification was real. The permission leak was not acceptable.
The second was a view rights change that would have exposed a read API to all users. My backend code runs server-side and blocks that API call, so in practice it would not have caused immediate harm. But I did not want an open API sitting there waiting for the day my server logic changed or someone found a route around it. I rejected it.
Both would have been invisible in normal testing. Neither would have triggered an immediate error. That is exactly the category of change that kills systems slowly.
What Git Actually Protected
After each validated change, I pushed to GitHub immediately. Not at the end of the day. After each change.
My normal habit is to push after 2-3 hours of coding. During this week I pushed every 15-20 minutes. If something had broken, the worst-case rollback would have been one session of work lost — not a week.
The paranoia from that earlier schema corruption was earned. The Git discipline made it manageable.
What I’m Still Figuring Out
Clean data is now in the system. Attendance and ops compliance are live and running without issues so far. Payroll logic is still under development. Whether the data is clean enough for that module will only become clear when edge cases surface during real use. I am not predicting the failure mode. I am watching for it.
Historical reconciliation has not started. Ten years of records still sit across the old systems. The plan from Log 3 holds — case by case, slowly. But I now have a clearer picture of what each case will cost. It will not be a weekend project.
The corrupted CSV records are no longer a concern. Before I caught the problem, the volume processed was close to 50 records. I deleted all of it and imported fresh data. The corruption did not survive into the live system.
The Raw Material
Two things you can apply before your migration starts.
Test your import pipeline against real data, not sample data. The edge cases that will break it — leading zeroes, commas in text, inconsistent date formats — only appear when you use actual records from your actual business. Sample data will not show you what your operations team’s Excel habits will do to your import logic.
Separate schema review from schema change. Before any AI-suggested modification touches your database, open a separate session. Give it access to your database documentation directly — not just your codebase. Assign it one mandate: analyse the risk of this specific change, score it, and explain what breaks. Give it no other goal. The first AI solves the problem. The second AI audits the solution. You hold the veto.
Aligned taxonomy likely saves time on migration. How much depends on how different your operations data is from machine-precision formatting. In my case, it reduced the problem. It did not remove it.