Skip to main content

Publishing History Archives

If you want to run a Full Validator, you need to set up your node to publish a history archive. You can host an archive using a blob store such as Amazon's S3 or Digital Ocean's spaces, or you can simply serve a local archive directly via an HTTP server such as Nginx or Apache. If you're setting up a Basic Validator, you can skip this section. No matter what kind of node you're planning to run, make sure to set it up to get history, which is covered in Environment Preparation.

One archive per node

Each node must publish to its own dedicated archive. Writing to the same archive from multiple nodes is not supported and will result in undefined behavior, potentially including data loss. If you're running multiple Full Validators (as Tier 1 organizations do), configure a separate archive for each — for example, history-a.example.com, history-b.example.com, and history-c.example.com.

Caching and History Archives​

The primary cost of running a validator will very likely be egress bandwidth. A crucial part of your strategy to manage those costs should be caching. You can significantly reduce these data transfer costs by using common caching techniques or a CDN. Three simple rules apply to caching the History archives:

  1. Do not cache the archive state file .well-known/stellar-history.json (Cache-Control: no-cache)
  2. Do not cache HTTP 4xx responses (Cache-Control: no-cache)
  3. Cache everything else for as long as possible (> 1 day)

Local History Archive Using nginx​

Here, we'll demonstrate how you can configure your node to store its history files in the local filesystem, and then publish that history using nginx for our webserver software.

First, you must add a history configuration stanza to your /etc/stellar/stellar-core.cfg configuration file.

[HISTORY.local]
get="cp /mnt/xvdf/stellar-core-archive/node_001/{0} {1}"
put="cp {0} /mnt/xvdf/stellar-core-archive/node_001/{1}"
mkdir="mkdir -p /mnt/xvdf/stellar-core-archive/node_001/{0}"

Then, you must run Stellar Core's new-hist command to create the local history archive.

sudo -u stellar stellar-core --conf /etc/stellar/stellar-core.cfg new-hist local

This command creates the history archive structure:

$ tree -a /mnt/xvdf/stellar-core-archive/
/mnt/xvdf/stellar-core-archive
└── node_001
    ├── history
    │   └── 00
    │       └── 00
    │           └── 00
    │               └── history-00000000.json
    └── .well-known
        └── stellar-history.json

6 directories, 2 files

Now that the history archive's file structure is ready, you can configure a virtual host in nginx to serve the local archive.

server {
  listen 80;
  root /mnt/xvdf/stellar-core-archive/node_001/;

  server_name history.example.com;

  # do not cache 404 errors
  error_page 404 /404.html;
  location = /404.html {
    add_header Cache-Control "no-cache" always;
  }

  # do not cache history state file
  location ~ ^/.well-known/stellar-history.json$ {
    add_header Cache-Control "no-cache" always;
    try_files $uri =404;
  }

  # cache entire history archive for 1 day
  location / {
    add_header Cache-Control "max-age=86400";
    try_files $uri =404;
  }
}

Amazon S3 History Archive​

Now, let's demonstrate a configuration where your node stores its history files using Amazon's S3 service. You can then publish that history using an Amazon S3 static site, or again use nginx for your webserver software. This time, using nginx, we'll include some proxy and CDN configuration, as well.

Start by adding a history configuration stanza to your /etc/stellar/stellar-core.cfg configuration file.

[HISTORY.s3]
get='curl -sf http://history.example.com/{0} -o {1}' # Cached HTTP endpoint
put='aws s3 cp --region us-east-1 {0} s3://bucket.name/{1}' # Direct S3 access

Then, you must run Stellar Core's new-hist command to create and initialize the S3 archive.

sudo -u stellar stellar-core --conf /etc/stellar/stellar-core.cfg new-hist s3

These S3 history files can be served with something as simple as an Amazon S3 static site.

Optionally, you may want to place a reverse proxy and CDN in front of the S3 static site (we'll use nginx for this example).

server {
  listen 80;
  root /srv/nginx/history.example.com;
  index index.html index.htm;

  server_name history.example.com;

  # use google nameservers for lookups
  resolver 8.8.8.8 8.8.4.4;

  # bucket.name s3 static site endpoint
  set $s3_bucket "bucket.name.s3-website-us-east-1.amazonaws.com";

  # do not cache 404 errors
  error_page 404 /404.html;
  location = /404.html {
    add_header Cache-Control "no-cache" always;
  }

  # do not cache history state file
  location ~ ^/.well-known/stellar-history.json$ {
    add_header Cache-Control "no-cache" always;
    proxy_intercept_errors on;
    proxy_pass  http://$s3_bucket;
    proxy_read_timeout 120s;
    proxy_redirect off;
    proxy_buffering off;
    proxy_set_header        Host            $s3_bucket;
    proxy_set_header        X-Real-IP       $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
  }

  # cache history archive for 1 day
  location / {
    add_header Cache-Control "max-age=86400";
    proxy_intercept_errors on;
    proxy_pass  http://$s3_bucket;
    proxy_read_timeout 120s;
    proxy_redirect off;
    proxy_buffering off;
    proxy_set_header        Host            $s3_bucket;
    proxy_set_header        X-Real-IP       $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
  }
}

Backfilling a History Archive​

Given the choice, it's best to configure your history archive prior to your node's initial sync with an existing network. That way your validator's history publishes as you join, and subsequently sync with, the network.

However, if you have not published an archive during the node's initial sync, the steps required to create a history archive for an existing validator — in other words, to upgrade a Basic Validator to a Full Validator — are quite straightforward. First, you'll need to stop your stellar-core instance:

systemctl stop stellar-core # modify this if not using systemctl

Then, add the history archive configuration to your node's /etc/stellar/stellar-core.cfg configuration file.

[HISTORY.local]
get="cp /mnt/xvdf/stellar-core-archive/node_001/{0} {1}"
put="cp {0} /mnt/xvdf/stellar-core-archive/node_001/{1}"
mkdir="mkdir -p /mnt/xvdf/stellar-core-archive/node_001/{0}"

Next, you must run Stellar Core's new-hist command to create and initialize the local archive. (This is done as the stellar user.)

sudo -u stellar stellar-core --conf /etc/stellar/stellar-core.cfg new-hist local

Now you can start your Stellar Core instance again:

systemctl start stellar-core # modify this if not using systemctl

As you allow your node to join the network again, you can watch it start publishing a few checkpoints to the newly created archive.

2019-04-25T12:30:43.275 GDUQJ [History INFO] Publishing 1 queued checkpoints [16895-16895]: Awaiting 0/0 prerequisites of: publish-000041ff

At this stage your validator is successfully publishing its history, which enables other users to join the network using your archive.

Complete History Archive​

The stellar-archivist command line tool scans, repairs, and mirrors history archives. Using the SDF package repositories, you can install it by running:

apt-get install stellar-archivist-rs

The tool is also published on crates.io.

For detailed usage, please run stellar-archivist-rs --help.

Scanning an archive​

scan is read-only. It reports files that are missing, and if passing in --verify it also checks file content integrity.

stellar-archivist-rs scan file:///mnt/xvdf/stellar-core-archive/node_001 --verify --report report.json
2026-08-20T17:09:21.958405Z  INFO Starting scan of file:///mnt/xvdf/stellar-core-archive/node_001
2026-08-20T17:09:21.960409Z  INFO Processing 32 checkpoints from 63 (0x0000003f) to 2047 (0x000007ff)
2026-08-20T17:09:21.961701Z ERROR Missing results/00/00/00/results-000000bf.xdr.gz
2026-08-20T17:09:21.974309Z ERROR Missing transactions/00/00/05/transactions-000005bf.xdr.gz
2026-08-20T17:09:21.977013Z ERROR Failed to process bucket/8c/e5/7e/bucket-8ce57e5f8c2ee574af879dde28cf87a0d936a5a1ea905b377409f275a4e73a1c.xdr.gz: Hash mismatch: got 49ec4c32c208f6eb53d0e847bc8a4819544a71ca04c20f47737ba0842f133370
2026-08-20T17:09:21.979973Z ERROR Missing scp/00/00/03/scp-0000037f.xdr.gz
2026-08-20T17:09:21.983537Z ERROR Missing scp/00/00/06/scp-0000067f.xdr.gz
2026-08-20T17:09:21.988195Z ERROR Missing history/00/00/06/history-000006ff.json
2026-08-20T17:09:22.002390Z ERROR Failed to process bucket/d6/2e/37/bucket-d62e37771e7ae305d981317d710cd9f6bdf254d16caa315955217deddaaaab05.xdr.gz: Hash mismatch: got 49ec4c32c208f6eb53d0e847bc8a4819544a71ca04c20f47737ba0842f133370
...
2026-08-20T17:09:22.020233Z  INFO Scan complete: 378 files found, 11 missing or corrupt
2026-08-20T17:09:22.020243Z ERROR 1 history file(s) missing or corrupt
2026-08-20T17:09:22.020248Z ERROR 1 transactions file(s) missing or corrupt
2026-08-20T17:09:22.020253Z ERROR 1 results file(s) missing or corrupt
2026-08-20T17:09:22.020257Z ERROR 6 bucket file(s) missing or corrupt
2026-08-20T17:09:22.020262Z  WARN 2 optional scp file(s) missing or corrupt
2026-08-20T17:09:22.020487Z  WARN Archive is incomplete: 11 issue(s) found
error: Archive issues found

--report writes the same findings as JSON:

{
  "version": 1,
  "well_known": null,
  "files": {
    "1471": ["transactions"],
    "1663": ["scp"],
    "1791": ["history"],
    "191": ["results"],
    "895": ["scp"]
  },
  "buckets": [
    "0e3f43118dc8761a94102e2f043f7994a0a12078220b2873dce162fd1d195a50",
    "344dd4abedf73ee3d65d07702e0a1a065b710fcfabc9202821ac50c16e5db898",
    "8ce57e5f8c2ee574af879dde28cf87a0d936a5a1ea905b377409f275a4e73a1c",
    "8e0eb1c3e797336973e5e7d3591b5bf6385de2d6d990d81a9732d76d61d62cf9",
    "d62e37771e7ae305d981317d710cd9f6bdf254d16caa315955217deddaaaab05",
    "dc1031942e58d472b8f7f9630250a296426c767afdca92fb687450adb87e1700"
  ],
  "checkpoints": [],
  "summary": {
    "succeeded": 378,
    "skipped": 0,
    "failed": 11,
    "retries": 0
  }
}
  • files maps a checkpoint ledger to the file types broken at that checkpoint.
  • buckets lists bucket hashes that are missing or failed their SHA-256 check.
  • checkpoints lists cross-file and hash chain failures, which are re-fetched whole.
  • well_known is the checkpoint to restore .well-known/stellar-history.json from, or null when it is healthy.

Repairing an archive​

repair fixes an archive in place by re-fetching broken and missing files from a known-good source — such as the SDF public history archive.

A report written by scan can be fed straight back in as a repair plan (with the --plan flag). Every file, bucket, and checkpoint it lists is re-fetched from the source; --verify validates the source content beforehand.

stellar-archivist-rs repair \
  https://history.stellar.org/prd/core-testnet/core_testnet_001 \
  file:///mnt/xvdf/stellar-core-archive/node_001 \
  --verify --plan report.json
2026-08-20T17:09:22.037862Z  INFO Starting repair from https://history.stellar.org/prd/core-testnet/core_testnet_001 to file:///mnt/xvdf/stellar-core-archive/node_001 with 32 workers
2026-08-20T17:09:22.235494Z  INFO Plan mode: applying report.json
2026-08-20T17:09:22.235617Z  INFO Retrying 11 failed file(s)
2026-08-20T17:09:22.266085Z  INFO Repair file retry completed: 23 files processed, 0 failed
2026-08-20T17:09:22.266112Z  INFO Repair checkpoint retry completed: 0 files processed, 0 failed

Repair runs in stages and reports each one separately: a main pass that audits the destination, a file retry that re-fetches everything recorded as broken, and a checkpoint retry for cross-file and chain failures. A repair with a pre-existing plan skips the main pass, because the plan already says what is broken.

A final scan confirms the archive has been repaired:

stellar-archivist-rs scan file:///mnt/xvdf/stellar-core-archive/node_001 --verify
2026-08-20T17:09:22.274206Z  INFO Starting scan of file:///mnt/xvdf/stellar-core-archive/node_001
2026-08-20T17:09:22.276206Z  INFO Processing 32 checkpoints from 63 (0x0000003f) to 2047 (0x000007ff)
2026-08-20T17:09:22.331753Z  INFO Progress: 32/32 checkpoints processed
2026-08-20T17:09:22.331790Z  INFO Scan complete: 395 files found, 0 missing or corrupt
tip

Omit --plan to scan and repair in a single command, without producing a report first.

tip

Add --dry-run to report what would be repaired without writing anything. Combined with --report, this writes a plan you can review and apply later.

Mirroring an archive​

mirror copies a source archive to a destination. Use it to onboard a fresh archive, or to backfill a range of history your validator did not publish itself.

stellar-archivist-rs mirror \
  https://history.stellar.org/prd/core-testnet/core_testnet_001 \
  file:///mnt/xvdf/stellar-core-archive/node_001 \
  --high 1023 --verify --report report.json
2026-08-20T17:09:23.266746Z  INFO Starting mirror from https://history.stellar.org/prd/core-testnet/core_testnet_001 to file:///mnt/xvdf/stellar-core-archive/node_001 with 32 workers
2026-08-20T17:09:23.664272Z  INFO No existing archive at destination; starting a fresh mirror
2026-08-20T17:09:23.664300Z  INFO Processing 16 checkpoints from 63 (0x0000003f) to 1023 (0x000003ff)
2026-08-20T17:09:23.850821Z  INFO Progress: 16/16 checkpoints processed
2026-08-20T17:09:23.850848Z  INFO Mirror completed: 196 files copied, 0 failed, 0 skipped
2026-08-20T17:09:23.850855Z  INFO No existing .well-known, creating new one at checkpoint 1023 (0x000003ff)
2026-08-20T17:09:23.851784Z  INFO Updated destination .well-known to checkpoint 1023 (0x000003ff)
tip

Mirroring is resumable. Running it again against the same destination with a higher --high picks up from the destination's current checkpoint and copies only what is missing, unless you pass --override.

Once the archive is complete, start your Stellar Core instance again.

systemctl start stellar-core

You should now have a complete history archive being written by your full validator. Congratulations!