{"id":3506,"date":"2026-10-02T17:00:43","date_gmt":"2026-10-02T22:00:43","guid":{"rendered":"https:\/\/xlinesoft.com\/blog\/?p=3506"},"modified":"2026-10-02T17:16:03","modified_gmt":"2026-10-02T22:16:03","slug":"process-large-csv-exports-in-background-batches","status":"publish","type":"post","link":"https:\/\/xlinesoft.com\/blog\/2026\/10\/02\/process-large-csv-exports-in-background-batches\/","title":{"rendered":"Process Large CSV Exports in Background Batches"},"content":{"rendered":"<p>Exporting hundreds of thousands of records during a normal page request can exceed PHP execution limits and leave the user waiting without any indication of progress.<\/p>\n<p>This PHPRunner tutorial moves large CSV exports into a scheduled worker. Users submit an export request, the worker processes a fixed number of records per run, and the List page shows progress until the ZIP file is ready to download.<\/p>\n<p><a href=\"https:\/\/xlinesoft.com\/blog\/wp-content\/uploads\/2026\/10\/export_queue.png\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/xlinesoft.com\/blog\/wp-content\/uploads\/2026\/10\/export_queue-1024x569.png\" alt=\"\" width=\"810\" height=\"450\" class=\"alignnone size-large wp-image-3512\" srcset=\"https:\/\/xlinesoft.com\/blog\/wp-content\/uploads\/2026\/10\/export_queue-1024x569.png 1024w, https:\/\/xlinesoft.com\/blog\/wp-content\/uploads\/2026\/10\/export_queue-600x334.png 600w, https:\/\/xlinesoft.com\/blog\/wp-content\/uploads\/2026\/10\/export_queue-768x427.png 768w, https:\/\/xlinesoft.com\/blog\/wp-content\/uploads\/2026\/10\/export_queue.png 1309w\" sizes=\"auto, (max-width: 810px) 100vw, 810px\" \/><\/a><br \/>\n<!--more--><\/p>\n<h3>1. Creating the export queue<\/h3>\n<p>Create an <strong>exports<\/strong> table and add it to the PHPRunner project. This example supports two export types and optional date limits.<\/p>\n<div class=\"my-syntax-highlighter\">\n<pre><textarea id=\"mshighlighter\" class=\"mshighlighter\" language=\"sql\" name=\"mshighlighter\" >\r\nCREATE TABLE exports (\r\n    id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,\r\n    created_at DATETIME NOT NULL,\r\n    date_from DATE NULL,\r\n    date_to DATE NULL,\r\n    status VARCHAR(20) NOT NULL,\r\n    total_records INT NOT NULL DEFAULT 0,\r\n    processed_records INT NOT NULL DEFAULT 0,\r\n    filename VARCHAR(255) NULL,\r\n    completed_at DATETIME NULL,\r\n    task_type VARCHAR(50) NOT NULL\r\n);<\/textarea><\/pre>\n<\/div>\n<p>On the Add page, show <strong>date_from<\/strong>, <strong>date_to<\/strong>, and <strong>task_type<\/strong>. Configure <strong>task_type<\/strong> as a Lookup wizard with the fixed values <strong>orders<\/strong> and <strong>transactions<\/strong>. Keep the worker-managed fields off the Add and Edit pages.<\/p>\n<h3>2. Queuing a new export<\/h3>\n<p>Open <strong>Exports Add page &#8211; Before record added<\/strong> and add:<\/p>\n<div class=\"my-syntax-highlighter\">\n<pre><textarea id=\"mshighlighter\" class=\"mshighlighter\" language=\"php\" name=\"mshighlighter\" >\r\n$allowedTypes = array(\"orders\", \"transactions\");\r\n\r\nif (!in_array($values[\"task_type\"], $allowedTypes, true)) {\r\n    $message = \"Select a valid export type.\";\r\n    return false;\r\n}\r\n\r\n$values[\"created_at\"] = date(\"Y-m-d H:i:s\");\r\n$values[\"status\"] = \"Pending\";\r\n$values[\"total_records\"] = 0;\r\n$values[\"processed_records\"] = 0;\r\n$values[\"filename\"] = null;\r\n$values[\"completed_at\"] = null;\r\n\r\nreturn true;<\/textarea><\/pre>\n<\/div>\n<p>The user only creates a queue record. No export data is generated during the Add request.<\/p>\n<h3>3. Adding the scheduled worker<\/h3>\n<p>Create <strong>cron.php<\/strong> in the generated application folder. This example uses MySQL syntax and processes up to 100,000 rows during each run. Change <strong>$recordsPerRun<\/strong> if the server needs smaller batches.<\/p>\n<div class=\"my-syntax-highlighter\">\n<pre><textarea id=\"mshighlighter\" class=\"mshighlighter\" language=\"php\" name=\"mshighlighter\" >\r\n<?php\r\n\r\nif (PHP_SAPI !== \"cli\") {\r\n    http_response_code(403);\r\n    exit(\"This worker must be run from the command line.\");\r\n}\r\n\r\nrequire_once __DIR__ . \"\/include\/dbcommon.php\";\r\n\r\nset_time_limit(300);\r\n\r\n$recordsPerRun = 100000;\r\n$progressUpdateInterval = 1000;\r\n\r\n$exportTypes = array(\r\n    \"orders\" => array(\r\n        \"table\" => \"orders\",\r\n        \"dateField\" => \"order_date\",\r\n        \"headers\" => array(\r\n            \"Order ID\",\r\n            \"Order Date\",\r\n            \"Customer\",\r\n            \"Amount\"\r\n        ),\r\n        \"fields\" => array(\r\n            \"id\",\r\n            \"order_date\",\r\n            \"customer\",\r\n            \"amount\"\r\n        )\r\n    ),\r\n    \"transactions\" => array(\r\n        \"table\" => \"transactions\",\r\n        \"dateField\" => \"transaction_date\",\r\n        \"headers\" => array(\r\n            \"Transaction ID\",\r\n            \"Transaction Date\",\r\n            \"Customer Name\",\r\n            \"Customer Email\",\r\n            \"Transaction Type\",\r\n            \"Amount\",\r\n            \"Status\",\r\n            \"Reference\"\r\n        ),\r\n        \"fields\" => array(\r\n            \"id\",\r\n            \"transaction_date\",\r\n            \"customer_name\",\r\n            \"customer_email\",\r\n            \"transaction_type\",\r\n            \"amount\",\r\n            \"status\",\r\n            \"reference\"\r\n        )\r\n    )\r\n);\r\n\r\n$rs = DB::Query(\r\n    \"SELECT * FROM exports \" .\r\n    \"WHERE status IN ('Pending', 'Processing') \" .\r\n    \"ORDER BY id LIMIT 1\"\r\n);\r\n$export = $rs->fetchAssoc();\r\n\r\nif (!$export) {\r\n    echo \"No exports to process.\\n\";\r\n    exit;\r\n}\r\n\r\n$exportId = (int)$export[\"id\"];\r\n$taskType = trim((string)$export[\"task_type\"]);\r\n\r\nif (!isset($exportTypes[$taskType])) {\r\n    DB::Update(\r\n        \"exports\",\r\n        array(\"status\" => \"Failed\"),\r\n        array(\"id\" => $exportId)\r\n    );\r\n    exit(\"Invalid task type.\\n\");\r\n}\r\n\r\n$definition = $exportTypes[$taskType];\r\n$sourceTable = $definition[\"table\"];\r\n$dateField = $definition[\"dateField\"];\r\n$alreadyProcessed = (int)$export[\"processed_records\"];\r\n\r\nDB::Update(\r\n    \"exports\",\r\n    array(\"status\" => \"Processing\"),\r\n    array(\"id\" => $exportId)\r\n);\r\n\r\n$where = array();\r\n\r\nif (!empty($export[\"date_from\"])) {\r\n    $where[] = DB::PrepareSQL(\r\n        $dateField . \" >= ':1'\",\r\n        $export[\"date_from\"] . \" 00:00:00\"\r\n    );\r\n}\r\n\r\nif (!empty($export[\"date_to\"])) {\r\n    $where[] = DB::PrepareSQL(\r\n        $dateField . \" <= ':1'\",\r\n        $export[\"date_to\"] . \" 23:59:59\"\r\n    );\r\n}\r\n\r\n$whereSql = count($where)\r\n    ? \" WHERE \" . implode(\" AND \", $where)\r\n    : \"\";\r\n\r\nif ($alreadyProcessed === 0) {\r\n    $countRs = DB::Query(\r\n        \"SELECT COUNT(*) AS cnt FROM \" . $sourceTable . $whereSql\r\n    );\r\n    $countData = $countRs->fetchAssoc();\r\n    $totalRecords = (int)$countData[\"cnt\"];\r\n\r\n    DB::Update(\r\n        \"exports\",\r\n        array(\"total_records\" => $totalRecords),\r\n        array(\"id\" => $exportId)\r\n    );\r\n} else {\r\n    $totalRecords = (int)$export[\"total_records\"];\r\n}\r\n\r\n$exportFolder = getabspath(\"exports\");\r\n\r\nif (!is_dir($exportFolder) && !mkdir($exportFolder, 0755, true)) {\r\n    DB::Update(\r\n        \"exports\",\r\n        array(\"status\" => \"Failed\"),\r\n        array(\"id\" => $exportId)\r\n    );\r\n    exit(\"Cannot create the exports folder.\\n\");\r\n}\r\n\r\n$csvFilename = \"export_\" . $exportId . \".csv\";\r\n$csvPath = $exportFolder . DIRECTORY_SEPARATOR . $csvFilename;\r\n$file = fopen($csvPath, $alreadyProcessed === 0 ? \"w\" : \"a\");\r\n\r\nif (!$file) {\r\n    DB::Update(\r\n        \"exports\",\r\n        array(\"status\" => \"Failed\"),\r\n        array(\"id\" => $exportId)\r\n    );\r\n    exit(\"Cannot open the export file.\\n\");\r\n}\r\n\r\nif ($alreadyProcessed === 0) {\r\n    fputcsv($file, $definition[\"headers\"]);\r\n}\r\n\r\n$sql = \"SELECT * FROM \" . $sourceTable . $whereSql;\r\n$sql .= \" ORDER BY id\";\r\n$sql .= \" LIMIT \" . (int)$recordsPerRun;\r\n$sql .= \" OFFSET \" . (int)$alreadyProcessed;\r\n\r\n$rs = DB::Query($sql);\r\n$processedThisRun = 0;\r\n\r\nwhile ($row = $rs->fetchAssoc()) {\r\n    $csvRow = array();\r\n\r\n    foreach ($definition[\"fields\"] as $field) {\r\n        $csvRow[] = $row[$field];\r\n    }\r\n\r\n    fputcsv($file, $csvRow);\r\n    $processedThisRun++;\r\n\r\n    if ($processedThisRun % $progressUpdateInterval === 0) {\r\n        DB::Update(\r\n            \"exports\",\r\n            array(\r\n                \"processed_records\" =>\r\n                    $alreadyProcessed + $processedThisRun\r\n            ),\r\n            array(\"id\" => $exportId)\r\n        );\r\n    }\r\n}\r\n\r\nfclose($file);\r\n\r\n$totalProcessed = $alreadyProcessed + $processedThisRun;\r\n$update = array(\"processed_records\" => $totalProcessed);\r\n\r\nif ($totalProcessed >= $totalRecords) {\r\n    $zipFilename = \"export_\" . $exportId . \"_\"\r\n        . bin2hex(random_bytes(12)) . \".zip\";\r\n    $zipPath = $exportFolder . DIRECTORY_SEPARATOR . $zipFilename;\r\n    $zip = new ZipArchive();\r\n\r\n    if (\r\n        $zip->open(\r\n            $zipPath,\r\n            ZipArchive::CREATE | ZipArchive::OVERWRITE\r\n        ) !== true\r\n    ) {\r\n        $update[\"status\"] = \"Failed\";\r\n        DB::Update(\"exports\", $update, array(\"id\" => $exportId));\r\n        exit(\"Cannot create the ZIP file.\\n\");\r\n    }\r\n\r\n    $zip->addFile($csvPath, $csvFilename);\r\n    $zip->close();\r\n\r\n    if (!file_exists($zipPath)) {\r\n        $update[\"status\"] = \"Failed\";\r\n        DB::Update(\"exports\", $update, array(\"id\" => $exportId));\r\n        exit(\"The ZIP file was not created.\\n\");\r\n    }\r\n\r\n    unlink($csvPath);\r\n    $update[\"status\"] = \"Completed\";\r\n    $update[\"filename\"] = $zipFilename;\r\n    $update[\"completed_at\"] = date(\"Y-m-d H:i:s\");\r\n} else {\r\n    $update[\"status\"] = \"Processing\";\r\n}\r\n\r\nDB::Update(\"exports\", $update, array(\"id\" => $exportId));\r\n\r\necho \"Processed \" . $processedThisRun . \" records.\\n\";<\/textarea><\/pre>\n<\/div>\n<p>The worker uses an allowlist in <strong>$exportTypes<\/strong>, so a value stored in <strong>task_type<\/strong> cannot become an arbitrary table name. Each run resumes at <strong>processed_records<\/strong>. The status changes to &#8216;Completed&#8217; only after the ZIP file has been created.<\/p>\n<p>The PHP Zip extension must be enabled. The web-server account also needs write permission to the generated application&#8217;s <strong>exports<\/strong> folder.<\/p>\n<h3>4. Scheduling the worker<\/h3>\n<p>Schedule the generated application&#8217;s <strong>cron.php<\/strong> with the PHP command-line executable. For example, a Linux cron job that runs every minute can use:<\/p>\n<div class=\"my-syntax-highlighter\">\n<pre><textarea id=\"mshighlighter\" class=\"mshighlighter\" language=\"\" name=\"mshighlighter\" >\r\n* * * * * \/usr\/bin\/php \/path\/to\/application\/cron.php<\/textarea><\/pre>\n<\/div>\n<p>Use the equivalent scheduled-task command on Windows. Do not configure overlapping runs. The script rejects web requests, so it cannot be started by visiting its URL.<\/p>\n<h3>5. Displaying progress on the List page<\/h3>\n<p>Add <strong>status<\/strong>, <strong>total_records<\/strong>, <strong>processed_records<\/strong>, <strong>filename<\/strong>, and <strong>completed_at<\/strong> to the Exports List page.<\/p>\n<p>Open <strong>Exports List page &#8211; JavaScript OnLoad<\/strong> and add:<\/p>\n<div class=\"my-syntax-highlighter\">\n<pre><textarea id=\"mshighlighter\" class=\"mshighlighter\" language=\"javascript\" name=\"mshighlighter\" >\r\nsetInterval(function() {\r\n    if ($(\"div.modal-dialog\").length) {\r\n        return;\r\n    }\r\n\r\n    location.reload();\r\n}, 10000);<\/textarea><\/pre>\n<\/div>\n<p>The page refreshes every ten seconds, but it does not interrupt a user who has an Add, Edit, or View popup open.<\/p>\n<h3>6. Adding the download link<\/h3>\n<p>Open the <strong>filename<\/strong> field&#8217;s View as settings, select <strong>Custom<\/strong>, and add:<\/p>\n<div class=\"my-syntax-highlighter\">\n<pre><textarea id=\"mshighlighter\" class=\"mshighlighter\" language=\"php\" name=\"mshighlighter\" >\r\nif ($data[\"status\"] === \"Completed\" && $value) {\r\n    $filename = basename($value);\r\n    $url = \"exports\/\" . rawurlencode($filename);\r\n    $caption = htmlspecialchars(\r\n        \"Download\",\r\n        ENT_QUOTES,\r\n        \"UTF-8\"\r\n    );\r\n\r\n    $value = '<a href=\"' . $url . '\" download>'\r\n        . $caption\r\n        . '<\/a>';\r\n} else {\r\n    $value = \"\";\r\n}<\/textarea><\/pre>\n<\/div>\n<p>Rebuild the project, submit an export request, and run the scheduled worker. The record should move from &#8216;Pending&#8217; to &#8216;Processing&#8217;, the processed count should increase after each batch, and the download link should appear after the ZIP file is complete.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Build a PHPRunner export queue that processes large CSV files in scheduled batches, reports progress, and provides a ZIP download when complete.<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,8],"tags":[],"class_list":["post-3506","post","type-post","status-publish","format-standard","hentry","category-php-category","category-tutorials"],"_links":{"self":[{"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/posts\/3506","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/comments?post=3506"}],"version-history":[{"count":2,"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/posts\/3506\/revisions"}],"predecessor-version":[{"id":3513,"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/posts\/3506\/revisions\/3513"}],"wp:attachment":[{"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/media?parent=3506"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/categories?post=3506"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xlinesoft.com\/blog\/wp-json\/wp\/v2\/tags?post=3506"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}