Before you start
Use a published ordinary survey and an enabled supplementary importer. Create a key with imports:write; replacement additionally needs imports:replace. The key’s group restrictions must include the destination and it must have no question restrictions. The creator must retain verified administrator access.
Set DATUMFORMS_PROJECT_ID, DATUMFORMS_GROUP_ID, DATUMFORMS_CSV_FILE, a persisted UUID in DATUMFORMS_UPLOAD_REQUEST_ID, and DATUMFORMS_IMPORT_OPTIONS as JSON. For example: {"mode":"upsert","question_refs":[456,457],"identity_question_ref":456,"expected_rows":2,"clear_blanks":false,"reference_location":false}. Replace question IDs and row count with the complete incoming file’s values.
Prepare the CSV
Use numeric question IDs as answer-column headers. The numeric headers must exactly match question_refs. Include group_id and a numeric project-member field_agent for new rows, or Unique_ID for an existing response (the numeric ID or its G prefix). Example structure:
field_agent,group_id,456,457
101,123,ITEM-001,Example item
101,123,ITEM-002,Second item
Obtain the member ID through members.list, the group ID through groups.list and question references through surveys.get. For optional reference coordinates, use lat, long and accuracy with reference_location: true; these do not replace captured GPS evidence.
Request sequence
- POST
/api/csv-import,action: prepare: sendproject_id,group_id,name(CSV filename),bytes(1–52,428,800),sha256(base64 SHA-256), stable UUIDrequest_idandoptions. - Retain
upload_id. If noprogress_idexists, PUT the unchanged file toupload_urlwith every returnedupload_headersvalue. - POST,
action: complete,upload_id. The server checks stored size, checksum, content type and encryption before queuing processing. - GET with
action=status&upload_id=.... Inspectstate,rows,phase,reviewandreview_token. Whenphaseisreview, approve the proposed insert/update/removal summary. - POST,
action: apply,upload_id,review_token. For replacement, also sendconfirm_removals: true. - Poll status until
state: completed. Verify destination rows and identities.
Options
| Field | Requirement |
|---|---|
mode |
upsert or replace |
question_refs |
1–500 positive question IDs |
expected_rows |
Complete nonblank input rows, 1–100,000 |
identity_question_ref |
An imported question identifying records; required for replacement |
clear_blanks |
true explicitly clears blank imported answer cells |
reference_location |
true imports paired coordinates as reference context |
Code example
Download the complete cURL uploader, JavaScript uploader or Python uploader. Run once to review. Rerun with the same file/UUID and DATUMFORMS_APPLY=yes to apply; replacement also requires DATUMFORMS_CONFIRM_REMOVALS=yes. These scripts use the uploader directly and do not create a parallel import service.
cURL
#!/bin/bash
set -euo pipefail
: "${DATUMFORMS_API_KEY:?Set key}"
: "${DATUMFORMS_PROJECT_ID:?Set project ID}"
: "${DATUMFORMS_GROUP_ID:?Set group ID}"
: "${DATUMFORMS_CSV_FILE:?Set CSV path}"
: "${DATUMFORMS_UPLOAD_REQUEST_ID:?Set persisted UUID}"
: "${DATUMFORMS_IMPORT_OPTIONS:?Set reviewed options JSON}"
API="${DATUMFORMS_API_ORIGIN:-https://app.datumforms.com}/api/csv-import"
call() { curl --fail-with-body -sS "$API" -H "Authorization: Bearer $DATUMFORMS_API_KEY" -H 'Content-Type: application/json' --data "$1"; }
checksum=$(openssl dgst -sha256 -binary "$DATUMFORMS_CSV_FILE" | openssl base64 -A)
bytes=$(wc -c < "$DATUMFORMS_CSV_FILE" | tr -d ' ')
prepared=$(call "$(jq -nc --argjson project "$DATUMFORMS_PROJECT_ID" --argjson group "$DATUMFORMS_GROUP_ID" --arg name "items.csv" --argjson bytes "$bytes" --arg checksum "$checksum" --arg request "$DATUMFORMS_UPLOAD_REQUEST_ID" --argjson options "$DATUMFORMS_IMPORT_OPTIONS" '{action:"prepare",project_id:$project,group_id:$group,name:$name,bytes:$bytes,sha256:$checksum,request_id:$request,options:$options}')")
upload=$(printf '%s' "$prepared" | jq -er .upload_id)
if [ "$(printf '%s' "$prepared" | jq -r '.progress_id // empty')" = '' ]; then
headers=()
while IFS= read -r header; do headers+=(-H "$header"); done < <(printf '%s' "$prepared" | jq -r '.upload_headers|to_entries[]|.key+": "+.value')
url=$(printf '%s' "$prepared" | jq -er .upload_url)
curl --fail-with-body -sS --request PUT "$url" "${headers[@]}" --data-binary "@$DATUMFORMS_CSV_FILE" || true
fi
call "$(jq -nc --arg id "$upload" '{action:"complete",upload_id:$id}')"
for ((attempt=0;attempt<180;attempt++)); do
status=$(curl --fail-with-body -sS --get "$API" -H "Authorization: Bearer $DATUMFORMS_API_KEY" --data-urlencode action=status --data-urlencode "upload_id=$upload")
phase=$(printf '%s' "$status" | jq -r '.phase // empty')
state=$(printf '%s' "$status" | jq -r .state)
if [ "$phase" = review ]; then
printf '%s\n' "$status" | jq '{upload_id,review}'
[ "${DATUMFORMS_APPLY:-no}" = yes ] || exit 0
token=$(printf '%s' "$status" | jq -er .review_token)
confirm=false; [ "${DATUMFORMS_CONFIRM_REMOVALS:-no}" != yes ] || confirm=true
call "$(jq -nc --arg id "$upload" --arg token "$token" --argjson confirm "$confirm" '{action:"apply",upload_id:$id,review_token:$token,confirm_removals:$confirm}')"
elif [ "$state" = completed ]; then
echo 'Import complete'; exit 0
elif [ "$state" = failed ] || [ "$state" = error ]; then
echo 'Import failed; inspect upload status' >&2; exit 1
fi
sleep 2
done
echo 'Processing continues. Resume with the same file and upload UUID.' >&2
exit 1
JavaScript
// Node.js 18+. A reviewed uploader job; no changes apply until explicitly approved.
import {readFile} from 'node:fs/promises';
import {createHash} from 'node:crypto';
const key=process.env.DATUMFORMS_API_KEY,API=(process.env.DATUMFORMS_API_ORIGIN||'https://app.datumforms.com')+'/api/csv-import';
const file=await readFile(process.env.DATUMFORMS_CSV_FILE);
const options=JSON.parse(process.env.DATUMFORMS_IMPORT_OPTIONS);
const project=Number(process.env.DATUMFORMS_PROJECT_ID),group=Number(process.env.DATUMFORMS_GROUP_ID);
const requestId=process.env.DATUMFORMS_UPLOAD_REQUEST_ID;
if(!key||!requestId||!project||!group)throw Error('Set key, project, group and persisted upload UUID');
async function call(method,input){const u=new URL(API);if(method==='GET')for(const[k,v]of Object.entries(input))u.searchParams.set(k,v);const r=await fetch(u,{method,headers:{Authorization:'Bearer '+key,'Content-Type':'application/json'},...(method==='POST'?{body:JSON.stringify(input)}:{})});const j=await r.json();if(!r.ok||j.success===false)throw Error(r.status+': '+j.message);return j;}
const prepared=await call('POST',{action:'prepare',project_id:project,group_id:group,name:'items.csv',bytes:file.length,sha256:createHash('sha256').update(file).digest('base64'),request_id:requestId,options});
if(!prepared.progress_id){
try{await fetch(prepared.upload_url,{method:'PUT',headers:prepared.upload_headers,body:file,redirect:'error'});}catch{/* Complete checks the stored bytes after an uncertain PUT. */}
}
await call('POST',{action:'complete',upload_id:prepared.upload_id});
let applied=false,finished=false;
for(let attempt=0;attempt<180;attempt++){
const status=await call('GET',{action:'status',upload_id:prepared.upload_id});
if(status.phase==='review'){
console.log(JSON.stringify({upload_id:prepared.upload_id,review:status.review},null,2));
if(process.env.DATUMFORMS_APPLY!=='yes')break;
await call('POST',{action:'apply',upload_id:prepared.upload_id,review_token:status.review_token,confirm_removals:process.env.DATUMFORMS_CONFIRM_REMOVALS==='yes'});applied=true;
} else if(status.state==='completed'){console.log('Import complete');finished=true;break;}
else if(['failed','error'].includes(status.state))throw Error('Import failed; inspect upload status');
await new Promise(resolve=>setTimeout(resolve,2000));
}
if(applied&&!finished)throw Error('Processing continues. Resume with the same file and upload UUID.');
Python
# Python 3.10+, standard library only.
import base64, hashlib, json, os, time, urllib.parse, urllib.request
API=os.environ.get('DATUMFORMS_API_ORIGIN','https://app.datumforms.com')+'/api/csv-import'
key=os.environ['DATUMFORMS_API_KEY']
with open(os.environ['DATUMFORMS_CSV_FILE'],'rb') as source: file=source.read()
def call(method,payload):
url=API+('?' + urllib.parse.urlencode(payload) if method=='GET' else '')
req=urllib.request.Request(url,method=method,data=json.dumps(payload).encode() if method=='POST' else None,headers={'Authorization':'Bearer '+key,'Content-Type':'application/json'})
with urllib.request.urlopen(req,timeout=60) as r: result=json.load(r)
if result.get('success') is False: raise RuntimeError(result.get('message'))
return result
prepared=call('POST',dict(action='prepare',project_id=int(os.environ['DATUMFORMS_PROJECT_ID']),group_id=int(os.environ['DATUMFORMS_GROUP_ID']),name='items.csv',bytes=len(file),sha256=base64.b64encode(hashlib.sha256(file).digest()).decode(),request_id=os.environ['DATUMFORMS_UPLOAD_REQUEST_ID'],options=json.loads(os.environ['DATUMFORMS_IMPORT_OPTIONS'])))
if not prepared.get('progress_id'):
req=urllib.request.Request(prepared['upload_url'],method='PUT',data=file,headers=prepared['upload_headers'])
try:
with urllib.request.urlopen(req,timeout=120) as r: r.read()
except OSError:
pass # Completion verifies the stored checksum and size.
call('POST',dict(action='complete',upload_id=prepared['upload_id']))
applied=False
finished=False
for attempt in range(180):
status=call('GET',dict(action='status',upload_id=prepared['upload_id']))
if status.get('phase')=='review':
print(json.dumps(dict(upload_id=prepared['upload_id'],review=status['review']),indent=2))
if os.environ.get('DATUMFORMS_APPLY')!='yes': break
call('POST',dict(action='apply',upload_id=prepared['upload_id'],review_token=status['review_token'],confirm_removals=os.environ.get('DATUMFORMS_CONFIRM_REMOVALS')=='yes'))
applied=True
elif status.get('state')=='completed':
print('Import complete')
finished=True
break
elif status.get('state') in ('failed','error'): raise RuntimeError('Import failed; inspect upload status')
time.sleep(2)
if applied and not finished: raise RuntimeError('Processing continues. Resume with the same file and upload UUID.')
Check the result and recover
A prepared upload is not an applied import. Keep the file and upload identity until completion. A 409 can indicate changed source rows, mismatched file identity or stale review; obtain a fresh review for changed data. A 410 means an unused upload expired. Resume with the original key; expiry or lost administrator access stops further processing. Managed research responses remain protected.