How We Fixed an Infinite Cron Loop in Production: The 5-Failure IMAP Threshold, Atomic Concurrency, and SRE Observability

In software engineering, there is an old truth every backend developer learns the hard way: every bug fix has the potential to introduce a new edge case. A few days ago, I published a post-mortem explaining how I fixed an aggressive database disconnect bug where a 1-second Google IMAP socket timeout would silently mark a merchant as disconnected. Removing that aggressive disconnect query solved the socket blip issue. But in solving it, we created an unintended trap: when a merchant's password was genuinely invalid, our background cron daemon got trapped in an infinite retry loop every 60 seconds. Here is the complete breakdown of how we diagnosed the runaway cron loop, engineered a resilient 5-failure threshold, eliminated concurrency race conditions, and achieved zero Sentry errors in production.

1. The Context: The Bug Behind the Bug

In our previous engineering post-mortem, I Was Using FamGateway When My Gmail Suddenly Disconnected, I detailed how our background IMAP daemon had a single line of aggressive code:

Whenever Google's IMAP server experienced a microsecond network socket timeout (e.g. [CLOSED] IMAP connection broken), the code immediately set gmail_connected = 0, locking the merchant out of real-time polling until they re-entered their password manually.

My initial fix was simple: delete that query entirely. If a connection failed, log the error to system_logs and let the background worker try again on the next polling cycle. For temporary network hiccups, this worked like a charm. But I had overlooked a critical scenario: what happens when the credentials are not temporarily glitching, but permanently wrong?

2. Incident Discovery: The 60-Second Loop in Production

During a routine server audit and merchant abuse check, I opened our production database to inspect recent error logs. What I saw was an alarming cascade of repetitive errors in system_logs:

Every single minute, exactly on schedule, our background cron was attempting to log into Google IMAP using an invalid 16-character Google App Password. Merchant USRAF1CAE322E had created test payment links and pending orders, but had entered an expired or revoked App Password.

Because the database never marked them as disconnected, our 1-minute cron daemon (api/cron.php) dutifully inspected their pending orders, saw gmail_connected = 1, and dispatched a curl request to api/imap-processor.php every 60 seconds.

This had three serious consequences:

• Log Pollution: Hundreds of redundant database rows were being written to system_logs every hour.
• Upstream Google Rate Limits: Hammering Google's IMAP endpoint (imap.gmail.com:993) with known bad credentials risks IP-level throttling or temporary bans from Google's security filters.
• Worker Thread Starvation: The background worker spent valuable TLS handshake cycles trying to authenticate a dead account instead of servicing active merchants.

3. The Architecture: The 5-Consecutive-Failure State Machine

We needed a design that achieved two seemingly contradictory goals:

1. Resilience to Network Blips: Do not disconnect an active merchant if Google IMAP drops a socket connection once or twice for a split second.
2. Isolation of Permanent Failures: Disconnect broken credentials promptly so they do not loop forever and degrade the platform.

The solution is a 5-consecutive-failure circuit breaker backed by a dedicated column: users.imap_fail_count INT NOT NULL DEFAULT 0.

If Google drops a packet on attempt 1, the failure count becomes 1. When the next polling cycle connects smoothly, the counter immediately resets to 0. A merchant never notices transient network drops. But if an App Password is truly revoked or deleted, it fails 5 times in a row, the account is automatically disconnected, and the cron loop terminates permanently.

4. Deep Concurrency & SRE Audit: 5 Edge Cases Fixed

Writing a counter seems simple on paper, but in high-throughput payment infrastructure processing concurrent webhook broadcasts and checkout status checks, edge cases quickly emerge. During our technical audit, we identified and eliminated 5 critical bugs:

A. Lockfile Premature Deletion Defeating Rate Limiting

In api/imap-processor.php, we maintain a 5-second file lock (sys_get_temp_dir() . '/famgw_imap_' . $userId . '.lock') to prevent checkout pages from flooding Google with duplicate IMAP requests.

However, our failure handler originally had this line:

When a customer is on the checkout page, the browser polls api/checkout-status.php every 2 seconds. If IMAP failed, the code immediately deleted the lockfile. When the frontend polled 2 seconds later, file_exists($lockFile) was false, so it slammed IMAP again! Under high traffic or multi-tab usage, this bypassed our 5-second rate limit completely.

The Fix: We removed @unlink($lockFile) from the failure branch. Now, even if IMAP fails, the 5-second lock file remains on disk with its timestamp. Any subsequent poll within 5 seconds receives an instant {"status": "locked"} without touching Google IMAP. Once 5 seconds elapse, the lock expires naturally.

B. Concurrency Race Conditions in Counter Increments

Initially, the failure counter was calculated in PHP memory:

If two concurrent requests (e.g. background cron and a customer's checkout tab) executed within milliseconds of each other, both read imap_fail_count = 0 from memory, and both wrote imap_fail_count = 1 back to MySQL, overwriting each other.

The Fix: We migrated the increment to a direct atomic query on the InnoDB engine:

InnoDB's row-level locking guarantees that every concurrent failure is serialized and counted accurately, eliminating race conditions entirely.

C. Hanging IMAP Socket Timeout Circuit Breakers

PHP's default IMAP socket timeout can take up to 60 to 120 seconds if Google experiences network latency or packet loss during the TLS handshake. Because our background processor enforces a 30-second execution limit (set_time_limit(30)), a slow socket handshake risked triggering an uncatchable fatal timeout.

The Fix: We explicitly set strict 8-second circuit breakers before invoking imap_open():

If Google does not respond within 8 seconds, the socket fails fast, logs the incident, and gracefully terminates.

D. Cron Batch Starvation via SQL INNER JOIN

In api/cron.php, our background sync scans pending orders and payment links created in the past 15 minutes to verify payments if customers closed their browser tabs:

Notice the flaw: the query selected LIMIT 10 pending users without checking if their Gmail was connected. If 10 pending orders belonged to merchants whose accounts were disconnected, all 10 slots were consumed. In PHP, the loop saw empty($u['gmail_connected']) and skipped them, but active merchants in the 11th and 12th positions were starved out of the verification cycle!

The Fix: We injected an INNER JOIN directly into the SQL query:

Disconnected merchants are excluded at the database index layer, guaranteeing that 100% of cron capacity is dedicated to active, connected merchants.

E. Status Guards on Completed & Expired Orders

When an order is completed, the customer's browser is often redirected to a confirmation or invoice page. During those final seconds, the browser fires one or two polling requests to api/checkout-status.php and api/verify-order.php.

Previously, these endpoints unconditionally fired a background curl to api/imap-processor.php regardless of order status. We added an explicit status guard:

If an order is already marked success or expired, zero background IMAP calls are made, cutting redundant server load dramatically.

5. Sentry SRE Observability: Zero Unresolved Issues

As part of FamGateway's commitment to enterprise-grade SRE telemetry, every runtime anomaly is tracked in real time via the official Sentry PHP SDK. When inspecting our Sentry project dashboard, we identified an error:

Sentry Issue FAMGATEWAY-G:
ErrorException: Notice: ob_end_clean(): Failed to delete buffer. No buffer to delete in api/imap-processor.php

In environments where fastcgi_finish_request() was absent (such as CLI cron executions), our output decoupling fallback called ob_end_clean() unconditionally. When no output buffer was active, PHP raised a notice, which Sentry's global error handler captured as an unhandled exception.

We patched the buffer flush routine with a safe level check:

We resolved the issue in Sentry, attached an official resolution note explaining the fix and architecture hardening, and verified that our Sentry dashboard now stands at 0 unresolved issues.

6. Live Production Verification: The Test Run

To verify the fix in production under real-world conditions, we ran an automated verification sequence against merchant USRAF1CAE322E, whose credentials were bad:

Our database logged the event cleanly:

When the merchant logs into Integrations, they are greeted by a clear dashboard alert informing them that automation was paused due to invalid credentials, with a prompt to enter a fresh 16-digit Google App Password. As soon as they enter valid credentials, the system tests the socket, sets gmail_connected = 1, and resets imap_fail_count = 0 automatically.

7. Why SRE Stability Matters for FamPay API Key Integrations & Payment Gateway Without GST

When developers integrate our FamPay API or search for a reliable payment gateway without GST, they are trusting FamGateway to power their livelihood. Whether you are running a Telegram bot, a SaaS web platform, or a digital store, every millisecond of background verification latency directly influences your customer checkout experience.

Here is why solving this cron retry loop directly supercharges our FamPay API and merchant ecosystem:

• Sub-3-Second Webhooks for FamPay API Key Integrations: When you generate a FamPay API key for your application, your customers expect instant fulfillment. By eliminating runaway cron loops, our IMAP verification daemons operate with zero queue backlog, guaranteeing that webhook dispatches fire in 3 to 5 seconds.
• Zero Processing Contention for Non-GST Merchants: Individual creators, students, and freelancers operating without a registered GST or current account cannot afford dropped payments. Our database-level INNER JOIN optimization guarantees that stale or disconnected accounts never consume cron batch slots, ensuring 100% of background verification bandwidth remains dedicated to active merchants.
• Uncompromising FamPay Payment Gateway Uptime: Combining atomic InnoDB failure increments with 8-second socket circuit breakers prevents Google IMAP socket drops from propagating across the server. Even during high-velocity traffic spikes on our dynamic UPI QR codes and payment links, the verification engine remains isolated, responsive, and resilient.

8. Key Engineering Takeaways

Building a 0% fee non-custodial payment gateway requires uncompromising reliability. Here are the core engineering lessons we established from this post-mortem:

1. Never Trust In-Memory State for Concurrency: In distributed systems or asynchronous web workers, use atomic database increments (UPDATE ... = ... + 1) with row-level locks instead of reading, incrementing in memory, and writing back.
2. Differentiate Transient from Permanent: Never treat a 1-second network socket blip as a revoked credential, but never let permanent credential failures loop endlessly. A calibrated failure threshold provides the best of both worlds.
3. Keep Cooldowns on Failure: If a rate-limiting lockfile exists to prevent server hammering, do not delete it when a call fails. Let the TTL protect your upstream APIs from rapid retries.
4. Filter Queues at the Database Layer: Don't fetch rows into memory and skip them with PHP if statements. Use SQL INNER JOINs and WHERE clauses so that broken or inactive records never starve active workloads.

Radical transparency and continuous optimization are why over 2,500 developers trust FamGateway for instant, direct-to-bank UPI automation in India.

Back to Homepage →