DatumForms.
Join the panel Sign in Create a survey

DATUMFORMS DOCUMENTATION

Import and replace supplementary data through the uploader API

Verify a CSV, review its proposed changes and apply the approved import.

For developers

Updated 15 November 2026

On this page

Before you start

Use a published ordinary survey and an enabled supplementary importer. Create a key with imports:write; replacement additionally needs imports:replace. The key’s group restrictions must include the destination and it must have no question restrictions. The creator must retain verified administrator access.

Set DATUMFORMS_PROJECT_ID, DATUMFORMS_GROUP_ID, DATUMFORMS_CSV_FILE, a persisted UUID in DATUMFORMS_UPLOAD_REQUEST_ID, and DATUMFORMS_IMPORT_OPTIONS as JSON. For example: {"mode":"upsert","question_refs":[456,457],"identity_question_ref":456,"expected_rows":2,"clear_blanks":false,"reference_location":false}. Replace question IDs and row count with the complete incoming file’s values.

Prepare the CSV

Use numeric question IDs as answer-column headers. The numeric headers must exactly match question_refs. Include group_id and a numeric project-member field_agent for new rows, or Unique_ID for an existing response (the numeric ID or its G prefix). Example structure:

field_agent,group_id,456,457
101,123,ITEM-001,Example item
101,123,ITEM-002,Second item

Obtain the member ID through members.list, the group ID through groups.list and question references through surveys.get. For optional reference coordinates, use lat, long and accuracy with reference_location: true; these do not replace captured GPS evidence.

Request sequence

  1. POST /api/csv-import, action: prepare: send project_id, group_id, name (CSV filename), bytes (1–52,428,800), sha256 (base64 SHA-256), stable UUID request_id and options.
  2. Retain upload_id. If no progress_id exists, PUT the unchanged file to upload_url with every returned upload_headers value.
  3. POST, action: complete, upload_id. The server checks stored size, checksum, content type and encryption before queuing processing.
  4. GET with action=status&upload_id=.... Inspect state, rows, phase, review and review_token. When phase is review, approve the proposed insert/update/removal summary.
  5. POST, action: apply, upload_id, review_token. For replacement, also send confirm_removals: true.
  6. Poll status until state: completed. Verify destination rows and identities.

Options

Field Requirement
mode upsert or replace
question_refs 1–500 positive question IDs
expected_rows Complete nonblank input rows, 1–100,000
identity_question_ref An imported question identifying records; required for replacement
clear_blanks true explicitly clears blank imported answer cells
reference_location true imports paired coordinates as reference context

Code example

Download the complete cURL uploader, JavaScript uploader or Python uploader. Run once to review. Rerun with the same file/UUID and DATUMFORMS_APPLY=yes to apply; replacement also requires DATUMFORMS_CONFIRM_REMOVALS=yes. These scripts use the uploader directly and do not create a parallel import service.

cURL

#!/bin/bash
set -euo pipefail
: "${DATUMFORMS_API_KEY:?Set key}"
: "${DATUMFORMS_PROJECT_ID:?Set project ID}"
: "${DATUMFORMS_GROUP_ID:?Set group ID}"
: "${DATUMFORMS_CSV_FILE:?Set CSV path}"
: "${DATUMFORMS_UPLOAD_REQUEST_ID:?Set persisted UUID}"
: "${DATUMFORMS_IMPORT_OPTIONS:?Set reviewed options JSON}"
API="${DATUMFORMS_API_ORIGIN:-https://app.datumforms.com}/api/csv-import"
call() { curl --fail-with-body -sS "$API" -H "Authorization: Bearer $DATUMFORMS_API_KEY" -H 'Content-Type: application/json' --data "$1"; }
checksum=$(openssl dgst -sha256 -binary "$DATUMFORMS_CSV_FILE" | openssl base64 -A)
bytes=$(wc -c < "$DATUMFORMS_CSV_FILE" | tr -d ' ')
prepared=$(call "$(jq -nc --argjson project "$DATUMFORMS_PROJECT_ID" --argjson group "$DATUMFORMS_GROUP_ID" --arg name "items.csv" --argjson bytes "$bytes" --arg checksum "$checksum" --arg request "$DATUMFORMS_UPLOAD_REQUEST_ID" --argjson options "$DATUMFORMS_IMPORT_OPTIONS" '{action:"prepare",project_id:$project,group_id:$group,name:$name,bytes:$bytes,sha256:$checksum,request_id:$request,options:$options}')")
upload=$(printf '%s' "$prepared" | jq -er .upload_id)
if [ "$(printf '%s' "$prepared" | jq -r '.progress_id // empty')" = '' ]; then
 headers=()
 while IFS= read -r header; do headers+=(-H "$header"); done < <(printf '%s' "$prepared" | jq -r '.upload_headers|to_entries[]|.key+": "+.value')
 url=$(printf '%s' "$prepared" | jq -er .upload_url)
 curl --fail-with-body -sS --request PUT "$url" "${headers[@]}" --data-binary "@$DATUMFORMS_CSV_FILE" || true
fi
call "$(jq -nc --arg id "$upload" '{action:"complete",upload_id:$id}')"
for ((attempt=0;attempt<180;attempt++)); do
 status=$(curl --fail-with-body -sS --get "$API" -H "Authorization: Bearer $DATUMFORMS_API_KEY" --data-urlencode action=status --data-urlencode "upload_id=$upload")
 phase=$(printf '%s' "$status" | jq -r '.phase // empty')
 state=$(printf '%s' "$status" | jq -r .state)
 if [ "$phase" = review ]; then
  printf '%s\n' "$status" | jq '{upload_id,review}'
  [ "${DATUMFORMS_APPLY:-no}" = yes ] || exit 0
  token=$(printf '%s' "$status" | jq -er .review_token)
  confirm=false; [ "${DATUMFORMS_CONFIRM_REMOVALS:-no}" != yes ] || confirm=true
  call "$(jq -nc --arg id "$upload" --arg token "$token" --argjson confirm "$confirm" '{action:"apply",upload_id:$id,review_token:$token,confirm_removals:$confirm}')"
 elif [ "$state" = completed ]; then
  echo 'Import complete'; exit 0
 elif [ "$state" = failed ] || [ "$state" = error ]; then
  echo 'Import failed; inspect upload status' >&2; exit 1
 fi
 sleep 2
done
echo 'Processing continues. Resume with the same file and upload UUID.' >&2
exit 1

Download import.sh

JavaScript

// Node.js 18+. A reviewed uploader job; no changes apply until explicitly approved.
import {readFile} from 'node:fs/promises';
import {createHash} from 'node:crypto';
const key=process.env.DATUMFORMS_API_KEY,API=(process.env.DATUMFORMS_API_ORIGIN||'https://app.datumforms.com')+'/api/csv-import';
const file=await readFile(process.env.DATUMFORMS_CSV_FILE);
const options=JSON.parse(process.env.DATUMFORMS_IMPORT_OPTIONS);
const project=Number(process.env.DATUMFORMS_PROJECT_ID),group=Number(process.env.DATUMFORMS_GROUP_ID);
const requestId=process.env.DATUMFORMS_UPLOAD_REQUEST_ID;
if(!key||!requestId||!project||!group)throw Error('Set key, project, group and persisted upload UUID');
async function call(method,input){const u=new URL(API);if(method==='GET')for(const[k,v]of Object.entries(input))u.searchParams.set(k,v);const r=await fetch(u,{method,headers:{Authorization:'Bearer '+key,'Content-Type':'application/json'},...(method==='POST'?{body:JSON.stringify(input)}:{})});const j=await r.json();if(!r.ok||j.success===false)throw Error(r.status+': '+j.message);return j;}
const prepared=await call('POST',{action:'prepare',project_id:project,group_id:group,name:'items.csv',bytes:file.length,sha256:createHash('sha256').update(file).digest('base64'),request_id:requestId,options});
if(!prepared.progress_id){
  try{await fetch(prepared.upload_url,{method:'PUT',headers:prepared.upload_headers,body:file,redirect:'error'});}catch{/* Complete checks the stored bytes after an uncertain PUT. */}
}
await call('POST',{action:'complete',upload_id:prepared.upload_id});
let applied=false,finished=false;
for(let attempt=0;attempt<180;attempt++){
 const status=await call('GET',{action:'status',upload_id:prepared.upload_id});
 if(status.phase==='review'){
   console.log(JSON.stringify({upload_id:prepared.upload_id,review:status.review},null,2));
   if(process.env.DATUMFORMS_APPLY!=='yes')break;
   await call('POST',{action:'apply',upload_id:prepared.upload_id,review_token:status.review_token,confirm_removals:process.env.DATUMFORMS_CONFIRM_REMOVALS==='yes'});applied=true;
 } else if(status.state==='completed'){console.log('Import complete');finished=true;break;}
 else if(['failed','error'].includes(status.state))throw Error('Import failed; inspect upload status');
 await new Promise(resolve=>setTimeout(resolve,2000));
}
if(applied&&!finished)throw Error('Processing continues. Resume with the same file and upload UUID.');

Download import.mjs

Python

# Python 3.10+, standard library only.
import base64, hashlib, json, os, time, urllib.parse, urllib.request
API=os.environ.get('DATUMFORMS_API_ORIGIN','https://app.datumforms.com')+'/api/csv-import'
key=os.environ['DATUMFORMS_API_KEY']
with open(os.environ['DATUMFORMS_CSV_FILE'],'rb') as source: file=source.read()
def call(method,payload):
    url=API+('?' + urllib.parse.urlencode(payload) if method=='GET' else '')
    req=urllib.request.Request(url,method=method,data=json.dumps(payload).encode() if method=='POST' else None,headers={'Authorization':'Bearer '+key,'Content-Type':'application/json'})
    with urllib.request.urlopen(req,timeout=60) as r: result=json.load(r)
    if result.get('success') is False: raise RuntimeError(result.get('message'))
    return result
prepared=call('POST',dict(action='prepare',project_id=int(os.environ['DATUMFORMS_PROJECT_ID']),group_id=int(os.environ['DATUMFORMS_GROUP_ID']),name='items.csv',bytes=len(file),sha256=base64.b64encode(hashlib.sha256(file).digest()).decode(),request_id=os.environ['DATUMFORMS_UPLOAD_REQUEST_ID'],options=json.loads(os.environ['DATUMFORMS_IMPORT_OPTIONS'])))
if not prepared.get('progress_id'):
    req=urllib.request.Request(prepared['upload_url'],method='PUT',data=file,headers=prepared['upload_headers'])
    try:
        with urllib.request.urlopen(req,timeout=120) as r: r.read()
    except OSError:
        pass # Completion verifies the stored checksum and size.
call('POST',dict(action='complete',upload_id=prepared['upload_id']))
applied=False
finished=False
for attempt in range(180):
    status=call('GET',dict(action='status',upload_id=prepared['upload_id']))
    if status.get('phase')=='review':
        print(json.dumps(dict(upload_id=prepared['upload_id'],review=status['review']),indent=2))
        if os.environ.get('DATUMFORMS_APPLY')!='yes': break
        call('POST',dict(action='apply',upload_id=prepared['upload_id'],review_token=status['review_token'],confirm_removals=os.environ.get('DATUMFORMS_CONFIRM_REMOVALS')=='yes'))
        applied=True
    elif status.get('state')=='completed':
        print('Import complete')
        finished=True
        break
    elif status.get('state') in ('failed','error'): raise RuntimeError('Import failed; inspect upload status')
    time.sleep(2)
if applied and not finished: raise RuntimeError('Processing continues. Resume with the same file and upload UUID.')

Download import.py

Check the result and recover

A prepared upload is not an applied import. Keep the file and upload identity until completion. A 409 can indicate changed source rows, mismatched file identity or stale review; obtain a fresh review for changed data. A 410 means an unused upload expired. Resume with the original key; expiry or lost administrator access stops further processing. Managed research responses remain protected.

Map fields · Replace lists · Monthly automation.

Need help with this task? Contact DatumForms. Include the article URL and the step you reached.

Tracking preferences

Essential functions stay on. Choose whether to allow optional tracking on this website. You can change your choices in the footer at any time.

Read our privacy policy