Skip to content

Consuming Sundial's AEMO raw-file archive (S3)

For external consumers of Sundial's AEMO market-data feed — prepared for Emma (Kinelli), July 2026.

Sundial retrieves AEMO files from the participant feed within ~10–15 seconds of each 5-minute boundary (falling back to public NEMWEB if the feed is unavailable) and publishes every retrieved file — byte-identical, original filename — to an S3 bucket, before any internal processing. This document is the consumer contract for that bucket.

Why S3 rather than a push into your network

Your design doc's requirements (§1) exist to protect the Kinelli network: private, narrow, write-only, revocable. Pulling from S3 satisfies them by never touching your network at all:

  • Private: outbound HTTPS from your side to S3 only. No inbound connection, no tailnet coupling, no shared node, no open port — nothing of Kinelli's is reachable by anyone.
  • Narrow: the credentials you receive can read one bucket. Nothing else.
  • Revocable: you can stop pulling at any time; Sundial can revoke the credential at any time. There is no standing link to tear down.
  • Content-bounded: run your allowlist in your fetch loop exactly as your sweeper does today (the bucket only ever contains files Sundial's own fetcher retrieved, currently dispatch + P5MIN products).

If, after using it, you'd prefer push semantics: S3 event notifications (delivered via an SQS queue you long-poll — still outbound-only from your side) are a small addition on Sundial's side. Ask and it will be built.

Bucket layout

dispatch/PUBLIC_DISPATCHIS_<datestamp>_<id>[_LEGACY].zip
dispatch/manifest.json
p5min/PUBLIC_P5MIN_<datestamp>_<id>[_LEGACY].zip
p5min/manifest.json
  • Files are the raw AEMO ZIPs, byte-identical, original names. Name order is time order (datestamps are Australia/Brisbane, YYYYMMDDHHmm).
  • Files accumulate, so the bucket doubles as a historical archive / backfill source. (The bucket is versioned; superseded versions expire after 7 days, current objects are retained.)
  • manifest.json per feed is a tiny pointer to the newest file:
{
  "latest": "PUBLIC_DISPATCHIS_202607281120_0000000529660042_LEGACY.zip",
  "key": "dispatch/PUBLIC_DISPATCHIS_202607281120_0000000529660042_LEGACY.zip",
  "source": "participant-ftp",
  "sizeBytes": 12345,
  "publishedAt": "2026-07-28T01:20:14.567Z"
}

source is participant-ftp or nemweb (which path retrieved that interval). Stale publishedAt is your feed-silence signal — the equivalent of your per-product silence alarms.

How to consume: poll the manifest

Poll <feed>/manifest.json (it's a few hundred bytes) and download the file it points to when latest changes. A 1-second poll costs roughly USD 0.03 per day; use conditional requests (If-None-Match with the last ETag) and the misses are 304-cheap.

Reference loop (Python, boto3):

import boto3, botocore, json, os, time

BUCKET = "<bucket name — supplied with the credentials>"
FEED = "dispatch"
DEST = "/volume1/aemo_drop/incoming"   # or wherever suits your pipeline

s3 = boto3.client("s3", region_name="ap-southeast-2")
etag = None

while True:
    try:
        kwargs = {"Bucket": BUCKET, "Key": f"{FEED}/manifest.json"}
        if etag:
            kwargs["IfNoneMatch"] = etag
        resp = s3.get_object(**kwargs)
        etag = resp["ETag"]
        manifest = json.load(resp["Body"])
        name = manifest["latest"]
        # your allowlist check on `name` goes here
        dest = os.path.join(DEST, name)
        if not os.path.exists(dest):
            s3.download_file(BUCKET, manifest["key"], dest + ".part")
            os.rename(dest + ".part", dest)   # atomic, like the SFTP plan
    except botocore.exceptions.ClientError as e:
        if e.response["Error"]["Code"] != "304":
            raise
    time.sleep(1)

Notes:

  • The .part → rename convention from your design carries over unchanged, so your existing sweeper (or an inotify trigger) can consume DEST as planned — or skip the intermediate directory entirely and write straight into your feed archive, since the allowlist already ran in the loop.
  • For backfill or gap-recovery, ListObjectsV2 with prefix dispatch/ gives the full archive in name (= time) order.

Latency expectations

Sundial's fetcher polls the participant feed at 1-second intervals; files land in S3 roughly 12–16 seconds after the 5-minute boundary (AEMO publishes ~10 s after it). With a 1-second manifest poll you have the file within a few seconds of that — comfortably ahead of the ~66 s dashboard and ~119 s NEMWEB public paths. If the participant feed fails, the fallback NEMWEB fetch publishes the same file 1–3 minutes after the boundary (source: "nemweb" tells you which happened).

Current content scope — read before finalising your allowlist

  1. The feed currently delivers only *_LEGACY.zip dispatch variants (AEMO subscription issue on Sundial's side, being chased). Your PUBLIC_DISPATCHIS_*.zip glob still matches, but the content is the legacy file format — validate a real sample against your next5m parser before go-live. When the subscription is fixed, current-format files appear automatically (Sundial's fetcher prefers them when both exist).
  2. Only DISPATCHIS is confirmed on the feed today; P5MIN is fetched when present. Your Tier 1 PREDISPATCH and all Tier 2 products (SCADA, rooftop PV, intermittent, demand, network) are not currently fetched. A full directory listing of the feed is available on request to settle what else exists; extending the fetch set is straightforward once agreed.

Credentials

Stuart will send, over a secure channel: bucket name, an access key ID and secret for a read-only IAM identity (GetObject/ListObject on this bucket only — verifiably nothing else), region ap-southeast-2. Rotation or revocation is a one-line change on Sundial's side; tell Stuart if you want a scheduled rotation.

Acceptance suggestions (mirroring your §8)

  1. Fetch dispatch/manifest.json → fresh publishedAt, sane latest.
  2. Download the file it points to → valid ZIP, correct AEMO content.
  3. Attempt PutObject / another bucket / any other AWS API → denied (proves the credential is read-only and single-bucket).
  4. Watch across a 5-minute boundary → new file appears within ~20 s.