Crash-Safe Submissions Against a Flaky External API

A backend I own submits regulated documents to an external governmental API. The API is slow, rate-limited, and occasionally drops connections, and every submission is legally significant: losing one or creating a duplicate both carry real consequences. A synchronous call with a retry was never going to be safe enough. The interesting work was owning the failure modes instead of the happy path.

The failure mode that forced the design

The naive shape is “call the API, mark the document sent.” That breaks in two concrete ways.

Crash between local state and the external call. Submission is logically two phases: mark the document as uploading, then perform the external call and mark it uploaded. If the process dies between those phases, the document is stranded in “uploading” forever. Operators cannot tell whether the external call landed.

Duplicate on retry. A timeout does not mean the external call failed. It often means the call succeeded and the response was lost. Retrying naively creates a second, legally distinct submission.

Both failures are invisible in the happy path and surface only when something crashes or the network misbehaves, which is exactly when they are hardest to diagnose.

A submission state machine

Every submission record carries an explicit status and the timestamp the upload started. Status moves through a small, named set:

// Status stored on every document record.
public enum SubmissionStatus {
  PENDING,         // created, not yet sent
  UPLOADING,       // phase 1: external call in flight
  FINISHING,       // phase 2 (online path): finish call in flight
  UPLOADED,        // external API accepted
  UPLOAD_FAILED,   // external API rejected, or recovery flagged a stuck upload
  FINISH_FAILED    // recovery flagged a stuck finish
}

Status transitions are the only way a record advances; there is no implicit “it probably worked.” Combined with an async executor, the HTTP call leaves the request thread, and the record’s status is the source of truth for where the submission actually is.

Recovering stuck uploads

The crash-between-phases case is the one a state machine alone cannot solve, because nothing is running to move the record forward. A scheduled job detects and resolves it:

// Recovers records stranded between phase 1 (UPLOADING) and the external call.
@Component
@RequiredArgsConstructor
@Slf4j
public class StuckSubmissionRecoveryJob {
  private static final Duration STUCK_THRESHOLD = Duration.ofMinutes(30);
  private static final List<SubmissionStatus> UPLOADING_STATES =
      List.of(SubmissionStatus.UPLOADING, SubmissionStatus.FINISHING);

  private final SubmissionRepository submissions;

  @Scheduled(fixedRate = 600_000) // every 10 minutes
  @SchedulerLock(name = "stuckSubmissionRecovery", lockAtLeastFor = "5m", lockAtMostFor = "9m")
  @Transactional
  public void recoverStuckUploads() {
    Instant threshold = Instant.now().minus(STUCK_THRESHOLD);
    List<SubmissionRecord> stuck =
        submissions.findByStatusInAndUploadStartedAtBefore(UPLOADING_STATES, threshold);

    for (SubmissionRecord record : stuck) {
      SubmissionStatus failed =
          record.getStatus() == SubmissionStatus.UPLOADING
              ? SubmissionStatus.UPLOAD_FAILED
              : SubmissionStatus.FINISH_FAILED;
      record.setStatus(failed);
      record.setUploadStartedAt(null);
      submissions.save(record);
    }
  }
}

Three details matter in that job.

A cluster-wide distributed lock. The backend runs several replicas. A plain @Scheduled job fires on every replica. ShedLock’s @SchedulerLock guarantees a single recovery run cluster-wide for the lock window, so two replicas cannot recover the same record at once.

A time-based threshold. A submission legitimately in flight must not be flagged as stuck. The 30-minute window is wide enough that anything still “uploading” past it has genuinely stalled.

Recovery resolves to a failed status for operator action. The job could auto-resubmit, but resubmission risks duplicates when the original call may have landed, so it leaves the decision to an operator. Blind retries are exactly what produces duplicates.

Idempotency and audit

Resubmission safety lives at the application boundary. Before any submission, the service checks for an existing submission record for that document and refuses a second one. Every attempt is recorded as a request row with its type and timestamp, and every failure is recorded with the external API’s error code and description. On success, the returned receipt is decoded and persisted to object storage, and the status advances to uploaded.

The effect is that for any document, an operator can answer three questions from the data: was it submitted, what did the external API say, and what the system stored. That is the minimum bar for a legally significant integration.

Honest gaps

The design is deliberately conservative in two places I would revisit.

Recovery does not yet record a synthetic error row. Recovered records move to a failed status, but the audit trail does not currently record that internal recovery produced the failure rather than the external API. The code calls this out as a follow-up, so an operator tracing a failure can still mistake a recovered crash for an API rejection.

Idempotency depends on local state. The duplicate guard checks our own submission records. If a crash happens after the external call succeeds but before the local status updates, a later manual retry could still duplicate. Closing that fully requires an idempotency key the external API honors, which it does not provide.

References

• ShedLock: distributed locking for scheduled tasks across replicas.
• Spring @Scheduled: the task scheduler the recovery job runs on.
• Apache HttpClient 5: the multipart upload transport for the external API.