GIS Institute

Blog & Tutorials

From Files to Pipelines: How GIS Data Stopped Living on Your Hard Drive

From Files to Pipelines: How GIS Data Stopped Living on Your Hard Drive

A
Administrator QGISI
Oct 05, 2026
252

For decades, doing GIS meant one thing above all: managing files. You downloaded a shapefile, saved it to a folder, opened it, and worked on your copy. If the data updated, you downloaded it again. If the dataset was 40 GB, you waited, and you found the disk space. The file on your drive was the data.

That model is quietly being replaced, not by a better file, but by a different idea of where data lives and how you reach it. It's one of the most consequential shifts in geospatial work in years, and it's worth understanding without the hype that usually surrounds anything with "cloud" in the name.


The file model and why it lasted

Let's be fair to the old way first, because it earned its long run.

Files are simple. A shapefile or a GeoTIFF sits on your disk, fully yours, readable offline, on a plane, in a field site with no signal. There's no subscription, no login, no dependency on someone else's server staying up. You can back it up, email it, archive it for twenty years and open it again. For an enormous amount of real GIS work, that simplicity is exactly right, and it isn't going anywhere.

The file model didn't survive this long by accident. It survived because, for a single analyst working on a bounded dataset, it's genuinely hard to beat.

Where it breaks

The problem isn't files. It's scale, and the modern reality of how much data there now is.

When a single satellite dataset is measured in terabytes, the "download a copy and work on it" model collapses. You can't download a copy. You don't have the disk, the time, or the bandwidth, and even if you did, the data would be out of date by the time the transfer finished. The old workflow assumes the dataset is small enough to hold. Increasingly, it isn't.

Three cracks show up as data grows:

  • Size. You need one county's worth of a continental dataset, but the file model makes you take all of it.
  • Freshness. A file is a snapshot. The moment it's saved, it starts aging, and everyone working from a copy is working from a slightly different past.
  • Duplication. The same dataset ends up copied onto hundreds of drives, each one a stale, redundant island.


What changed: the data comes to you

The shift is deceptively simple to state: instead of downloading the whole dataset and then working on it, you query the dataset where it lives and pull only the bytes you actually need.

This is what "cloud-native" formats make possible, and the key thing to understand is that they aren't exotic new file types you have to learn from scratch. Most are clever reorganizations of formats you already know, designed so that a program can reach into a file sitting on a web server and grab just the relevant slice — using ordinary HTTP range requests, no specialized GIS server required. A few worth knowing by name:

  • Cloud-Optimized GeoTIFF (COG): a regular GeoTIFF, internally reorganized so you can stream just the area and resolution you need from cloud storage. Crucially, it's backwards-compatible: older tools that read GeoTIFF can still read a COG.
  • GeoParquet: geospatial vector data built on Apache Parquet, the columnar format that became the backbone of the modern analytics stack. Columnar storage means dramatically faster queries on large tables, and it plugs into query engines like DuckDB.
  • PMTiles: a single-file archive of map tiles you can host on any plain web server, no tile-server software needed.
  • COPC and FlatGeobuf: cloud-optimized point clouds and streaming-friendly vector data, respectively.

The common thread: the data stays in one authoritative place, and you fetch slices of it on demand. No full download. No stale copy. One source of truth.


From "files" to "pipelines"

Once data stops being something you hold and becomes something you query, the way you build work changes too.

Instead of a folder of files and a sequence of manual steps, you start building pipelines, repeatable chains that pull the current data from its source, process it, and produce an output, without a human downloading anything in between. Run the pipeline today, run it next month, and it fetches whatever the source holds now. The analyst's job shifts from managing copies to designing flows.

This is also exactly why the job postings we analyzed in an earlier issue keep asking for cloud platform familiarity, SQL, and Python alongside core GIS. Those aren't unrelated buzzwords. They're the toolkit for working with data that lives somewhere other than your hard drive.

The honest tradeoffs

Here's where a newsletter that promised "no hype" has to earn it. Cloud-native is not a strict upgrade. It's a different set of tradeoffs, and pretending otherwise would be doing you a disservice.

  • You trade independence for freshness. Streaming from a source means depending on that source, and on your network. The file on your disk works in a field site with no signal. A cloud query does not.
  • You trade one-time cost for ongoing cost. Cloud storage and data egress are recurring bills. A file you downloaded once is paid for.
  • You trade simplicity for capability. Setting up a pipeline is more conceptual overhead than opening a shapefile. For a small, one-off task, that overhead isn't worth it, the humble file still wins.
  • Immutability cuts both ways. Formats like PMTiles and GeoParquet are read-optimized and effectively immutable, brilliant for distribution, wrong for a dataset you need to edit transactionally. For that, a database like PostGIS is still the right tool.

The good news for anyone reading this from a QGIS seat: modern QGIS already handles these formats. COG, COPC, FlatGeobuf, GeoParquet, and PMTiles can be streamed directly, and the ecosystem, GDAL underneath, plus plugins like the GeoParquet Downloader, keeps closing the gap. You don't need to leave the tool you know to work this way.

The takeaway

The file isn't dead. For a solo analyst on a bounded, offline, one-off task, downloading a shapefile is still often the right call, and anyone who tells you otherwise is selling something.

But the center of gravity has moved. As datasets outgrow the drives meant to hold them, the winning approach is less about having the data and more about reaching it, querying an authoritative source for exactly the slice you need, and building repeatable pipelines instead of folders of aging copies.

The skill that matters now isn't managing files. It's understanding when the data should come to you, when you should still go to it, and knowing the difference. As always, the tooling is only as good as the judgment behind it.

Share this post
My Cart
0 Items

Your cart is empty

Start Browsing