The Complete lakeFS Guide

This is part one of three. It takes you from never having seen lakeFS to using it confidently on a real dataset. By the end you can start a lakeFS server on your laptop, create a repository, put files into it, branch it, change data safely on the branch, compare the change with the original, merge it or throw it away, and tag the result so you can find it again in six months. You will also understand which parts of lakeFS are free, which are not, and what changed in the release this guide was checked against, lakeFS 1.87.

Each section ends with a Try it task. Do them as you go. Version control for data is an idea that feels abstract until you have watched a branch appear instantly next to a dataset that holds a few gigabytes, so please run the commands rather than only reading them.

What lakeFS is and what you can do with it

lakeFS puts a Git-like layer of version control on top of an object store. An object store is the kind of storage that holds files as "objects" inside "buckets": Amazon S3, Google Cloud Storage, Azure Blob Storage, MinIO, and other storage that speaks the S3 protocol. Your data stays in your bucket, in the format you already use. lakeFS does not convert it, does not hold a second copy of it, and does not care whether the files are Parquet, CSV, images, audio, model checkpoints, or Delta and Iceberg table files. What lakeFS adds is a set of pointers and metadata that say which version of which file belongs to which branch and which commit.

The best way to feel why this matters is to picture a data team on a normal Tuesday. A nightly job rewrites a folder of Parquet files that a dashboard and two machine learning models read from. One night the job has a bug, and the folder now holds half-written files and wrong numbers. Everything downstream is broken, nobody is sure what the folder looked like yesterday, and the only "backup" is a copy somebody made three weeks ago. With lakeFS the nightly job writes to a branch, a check runs against the branch, and only if the check passes is the branch merged into main. Readers who point at main never see the broken state. If something slips through anyway, you revert to the previous commit, and the folder is back exactly as it was.

In practice people use it for four things. First, isolated experiments: give every engineer or every pipeline run a private branch of the whole dataset, at almost no cost. Second, safe publishing: write new data on a branch, validate it, then merge it in one atomic step so readers see either the old version or the new one and never a half-finished mix. Third, reproducibility: a commit is an immutable snapshot, so a model trained on lakefs://my-repo/3f9a.../data/ can be retrained on exactly the same data a year later. Fourth, fast recovery: undoing a bad change is a metadata operation that takes moments, however large the data is.

Everything in this guide is checked against lakeFS v1.87.0, released on 22 September 2026. You will see the word "Community" a lot. It names the free edition you download and run yourself. The commercial editions are called Team, Enterprise and Cloud, and they add things such as multiple users, permissions and single sign-on. Section four explains the difference, because one change in 1.87 affects almost every beginner tutorial you will find on the internet.

The problem lakeFS solves, and what came before

Code has had good version control for decades. You branch, you commit, you review a diff, you merge, and if you broke something you revert. Data mostly has not had any of that. A folder in a bucket is a single mutable thing: when you write to it, the old contents are gone, everybody who reads it sees the change at once, and there is no record of what it looked like before.

People have worked around this in several ways, and it helps to know them because they explain what lakeFS is competing with. The simplest is copying: before a risky change, copy the folder to a new prefix such as data_backup_2026_09_30/. This works until the data is large. Copying several terabytes is slow and expensive, nobody remembers which copy is which, and the copies are never cleaned up. The second workaround is naming conventions: writing each run into data/run=2026-09-30/ and keeping a pointer file that says which run is current. This is better, but the convention lives in people's heads, there is no atomic way to switch a whole dataset from one run to the next, and different teams invent different conventions.

A third approach is the versioning feature of the bucket itself, such as S3 object versioning. It keeps old versions of single objects, which helps if you delete one file by accident. But it has no concept of a consistent set of files at one moment. If a dataset is a thousand Parquet files, bucket versioning cannot tell you "the thousand files as they were last Monday at 9 a.m.", and it offers no way to try a change in isolation and then publish it as one unit.

A fourth approach is to use a table format such as Delta Lake or Apache Iceberg, which keeps a transaction log and supports time travel. These are excellent, and lakeFS works alongside them, but they version one table at a time. Real pipelines often touch many tables, raw files, images, and model artifacts together. lakeFS versions the whole repository as one unit, whatever the files are, so a single commit can capture the raw input, the cleaned tables, and the trained model that came from them.

What makes lakeFS practical is that it does not copy data to do any of this. Creating a branch is a metadata-only, zero-copy operation that takes constant time, so a branch of a repository holding a hundred million objects is as cheap as a branch of one holding ten. The sizing guide in the lakeFS documentation puts branch operations under about 30 milliseconds at the 90th percentile. That is what turns "give every pipeline run its own branch" from a nice idea into something you can actually do.

The mental model: repositories, branches, commits and tags

If you know Git, most of lakeFS will feel familiar, and the differences are worth noticing. If you do not know Git, the nouns below are all you need. There are only a handful.

An object is a file managed by lakeFS. The bytes live in the underlying object store, and lakeFS keeps a pointer and some metadata. A repository is a named collection of objects, together with all their branches, commits and tags. Repository names must start with a lowercase letter or a number, may contain only lowercase letters, numbers and hyphens, and must be between 3 and 63 characters long. So my-repo and sales-2026 are fine and My_Repo is not.

Every repository is given a storage namespace when you create it. This is the location in your object store where all of the repository's data lives, for example s3://my-bucket/my-repo. Committed data and not-yet-committed data are both written there, and lakeFS keeps its own metadata files under a _lakefs/ folder inside it. Two repositories may not share a namespace. The underlying location of a particular object is called its physical address, and lakeFS maps the logical paths you use to these physical addresses. Several logical paths, across different branches and commits, can point at one physical object, which is exactly why branching costs nothing: a new branch points at the same physical objects as the branch it came from. Objects that lakeFS has written are never modified in place.

A branch is a pointer to a commit, plus a set of uncommitted changes, also called staged changes. When you upload a file to a branch it becomes an uncommitted change on that branch and is visible to anyone reading that branch, but it is not yet part of history. A commit is an immutable snapshot of the entire repository at one moment, with a committer, a timestamp, a message, and optional key-value metadata. Each commit has a commit ID, which is a long hexadecimal digest. You can refer to a commit by a unique prefix of it. The default branch of a new repository is called main.

A tag is an immutable, human-friendly name for one commit, such as v1.0. Branches move as you commit; tags stay put. Use a tag when you want to say "this is the exact data the paper used".

A ref is the general name for any of those things: a branch, a tag, or a commit ID. lakeFS also understands Git-style suffixes on a ref. main~1 means the parent of the current head of main, and main~2 means the grandparent. You will use these when comparing "now" against "a bit ago".

Every object has a lakeFS URI with a fixed shape:

lakefs://<repository>/<ref>/<path>

So lakefs://my-repo/main/data/x.csv is the file data/x.csv on the main branch of my-repo, and lakefs://my-repo/v1.0/data/x.csv is the same path as it was at tag v1.0. If you have seen older tutorials that write lakefs://repo@branch/path, that @ syntax was replaced by a slash long ago, and using it gives a "not found" error.

Finally, a merge combines the changes from one ref (the source) into a branch (the destination). lakeFS does a three-way merge against the common ancestor. The important beginner fact is that conflicts are detected per whole object: lakeFS never tries to merge the inside of two different versions of a CSV. If the same file was changed differently on both sides, or changed on one side and deleted on the other, that is a conflict and you have to choose.

Editions and the 1.87 licence change

Before you install anything, you should know what you are installing, because 1.87 changed the ground rules and most tutorials were written before it.

lakeFS Community is the edition you can download and run yourself, and it is what this guide uses. It is free to use under the Business Source License 1.1 from version 1.87 onward. In plain words, the licence lets you use the unmodified software in production, but only for your own organisation's internal business purposes. You may not host it as a service for third parties, whether they pay or not. Changing behaviour through the documented configuration, flags, plug-ins and interfaces does not count as modifying it. The licence also says that each version converts to Apache 2.0 four years after that version was first published. If you are a student or a data engineer at a company using lakeFS for its own data, you are well inside those terms. If you are planning to build a product that offers lakeFS to customers, read the LICENSE file in the repository and ask someone who can give legal advice. Some pages of the official FAQ still say "Apache 2.0"; the licence file and the editions page are the current word.

The second change that matters to a beginner is about users. In 1.87, the Community edition runs with a single administrator user who has a single set of credentials: one access key ID and one secret access key. The earlier support for many users, groups, policies and a pluggable IAM service was removed. The web pages that managed users and groups are gone, and commands such as lakectl auth groups return a "not implemented" error against a Community server. Multiple users, fine-grained permissions, single sign-on and audit features belong to the commercial editions (Team, Enterprise and Cloud).

For a beginner, this has a practical meaning. Everyone and everything that talks to your Community server uses the same key pair, so treat that key pair like a password to the whole system, and do not paste it into notebooks you will share. If you are on a team that needs different people to have different permissions on different repositories, that is the moment to look at the commercial editions rather than trying to work around it.

Installing lakeFS and checking the setup

There are several ways to get lakeFS, and which one you choose depends on what you already have. For learning, I recommend the route the current documentation recommends first: the Python quickstart. For everything else, Docker is the most portable, Homebrew is convenient on macOS and Linux, and plain binaries work everywhere.

Two programs come in every installation. lakefs is the server, the thing that keeps running and answers requests. lakectl is the command-line client that you type commands into, and it talks to a running server over HTTP. Beginners often mix these up, so keep the distinction in mind: lakefs starts lakeFS, and lakectl uses it.

Option one: the Python quickstart

If you have Python 3.10 or newer, this is the shortest path on Linux, macOS and Windows:

pip install lakefs
python -m lakefs.quickstart

The second command downloads a lakefs server and a lakectl client that match the installed SDK version, places them in your virtual environment's bin/ folder (or Scripts\ on Windows, or ~/.lakefs/bin if you are not in a virtual environment), and starts the server in quickstart mode. When it is ready you see a line like lakeFS running in quickstart mode. Login at http://127.0.0.1:8000/. The quickstart uses fixed credentials that are the same for everyone, which is why it must never be used for real data:

Access Key ID:     AKIAIOSFOLQUICKSTART
Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Quickstart mode stores its metadata in a local embedded database and its data on your local disk, so it needs no cloud account, no bucket and no PostgreSQL. That is exactly what you want for learning.

Option two: Docker

If you already know Docker (see the Docker guide if you do not), you can run the server with one command. This is the equivalent of the quickstart:

docker run --name lakefs --pull always --rm -p 8000:8000 treeverse/lakefs:1.87.0 run --quickstart

The -p 8000:8000 publishes the server's port, and everything after the image name is passed to the lakefs program, so run --quickstart is the same as running lakefs run --quickstart on the host. If you want the data to survive a restart, run it with a local database, a local blockstore (the place objects are stored), a mounted folder, and a secret used to encrypt stored credentials:

docker run --name lakefs -p 8000:8000 \
  -v /path/on/host:/home/lakefs/lakefs \
  -e LAKEFS_DATABASE_TYPE=local \
  -e LAKEFS_BLOCKSTORE_TYPE=local \
  -e LAKEFS_AUTH_ENCRYPT_SECRET_KEY="change-me-to-a-long-random-string" \
  treeverse/lakefs:1.87.0 run

Notice the naming rule for environment variables: every configuration key becomes LAKEFS_ followed by the key in upper case, with dots replaced by underscores. So auth.encrypt.secret_key becomes LAKEFS_AUTH_ENCRYPT_SECRET_KEY. Inside a container, localhost means the container itself, so a client running on your laptop reaches the server at http://localhost:8000, but a second container would not.

Option three: Homebrew or a downloaded binary

On macOS and Linux, Homebrew installs both programs at once:

brew tap treeverse/lakefs
brew install lakefs
lakefs run --local-settings

--local-settings uses a local database and local storage, like the quickstart, but without the fixed credentials: the first time you open the web page, a setup wizard asks you to create your administrator. Alternatively, download lakeFS_1.87.0_<OS>_<arch>.tar.gz (or the .zip for Windows) from the GitHub releases page, check it against checksums.txt, unpack it, and put lakefs and lakectl on your PATH. Windows binaries exist, but the sizing guide says they are not rigorously tested and that you should not run lakeFS in production on Windows. For learning on Windows, use the Python quickstart or Docker with WSL2.

Checking that it works

Whichever way you started the server, open a second terminal and run these checks:

lakefs --version
lakectl --version
curl -i http://localhost:8000/_health

Both version commands should print 1.87.0, and the health endpoint should return HTTP 200. That _health URL is what load balancers use later in production. You can also open http://localhost:8000/ in a browser, where the web interface lives. The server banner in its terminal reads lakeFS 1.87.0 - Up and running (^C to shutdown)....

Connecting lakectl to your server

The server is running, but lakectl does not yet know where it is or who you are. It reads this from a small YAML file in your home directory. The simplest way to create that file is the interactive command:

lakectl config

It asks for three things: the access key ID, the secret access key, and the server endpoint URL. For the quickstart, type the two fixed credentials shown earlier and http://localhost:8000 as the endpoint. The command writes ~/.lakectl.yaml. You can have lakectl read a different file with --config (or -c), or with the environment variable LAKECTL_CONFIG_FILE. You can also skip the file entirely and use environment variables, which is handy in scripts and containers:

export LAKECTL_CREDENTIALS_ACCESS_KEY_ID="AKIAIOSFOLQUICKSTART"
export LAKECTL_CREDENTIALS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
export LAKECTL_SERVER_ENDPOINT_URL="http://localhost:8000"

The configuration file itself looks like this:

credentials:
  access_key_id: AKIAIOSFOLQUICKSTART
  secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
server:
  endpoint_url: http://localhost:8000

Now test the connection. These two commands ask the server who you are and then run a broader self-check of configuration, credentials and connectivity:

lakectl identity
lakectl doctor
lakectl repo list

On a fresh server repo list prints an empty table, and that is success: it means authentication worked and there is simply nothing there yet. If you see an authorization error, the key pair is wrong. If you see a connection error, the server is not running or the endpoint is wrong.

If you started the server with --local-settings or from a Docker image with a local database, rather than the quickstart, you create your own administrator the first time you open the web page. The setup wizard asks for a user name and some contact preferences, and it shows you the access key and secret once. Save them somewhere safe immediately, and notice that the wizard also offers a ready-made .lakectl.yaml. The same thing can be done without a browser using lakefs setup --user-name admin, which prints the new credentials.

Your first repository, step by step

The quickest way to get a repository with data in it is the sample repository that the web interface creates for you. Open http://localhost:8000/, log in with your credentials, and click Create Sample Repository. It creates a repository with example data and branches, which is perfect for clicking around. But the skill you need is doing it yourself from the command line, so we will build a small repository from scratch. It has a clear shape, and you will see every step.

Step one: create the repository

A repository needs a name and a storage namespace. For the local quickstart server, the namespace uses the local:// scheme. On a real deployment it would be an s3://, gs:// or Azure https:// address of a bucket you own. The scheme must match the blockstore the server was started with.

lakectl repo create lakefs://my-repo local://my-repo

If you are running against a real S3-backed server, the equivalent is:

lakectl repo create lakefs://my-repo s3://my-bucket/my-repo

The command prints a confirmation and the default branch name, main. Now check that it exists:

lakectl repo list

Step two: put a file in

Create a tiny dataset on your laptop and upload it to main. The -s flag says where the source is on your disk.

mkdir -p ~/lakefs-demo && cd ~/lakefs-demo
printf 'id,name,score\n1,Amal,71\n2,Bassem,64\n3,Carla,88\n' > scores.csv
lakectl fs upload lakefs://my-repo/main/data/scores.csv -s ./scores.csv

The object is now on main, but it is an uncommitted change. Ask lakeFS what has changed on the branch:

lakectl diff lakefs://my-repo/main

The output lists one added object, data/scores.csv. This is the same idea as git status. Nothing is in history yet. Take the snapshot with a commit, and always give it a message:

lakectl commit lakefs://my-repo/main -m "Add first scores file"

The command prints the new commit ID and message. An empty message is refused unless you add --allow-empty-message, which is a good default to keep.

Step three: look around

Three small commands let you inspect what is in the repository. Listing shows what objects exist, cat prints an object's contents, and stat shows its metadata such as size and checksum:

lakectl fs ls lakefs://my-repo/main/data/
lakectl fs cat lakefs://my-repo/main/data/scores.csv
lakectl fs stat lakefs://my-repo/main/data/scores.csv
lakectl log lakefs://my-repo/main

lakectl log shows the history, newest first. You should see one commit, yours. You have just made a repository, added data, and recorded a version.

Branching: the feature that makes lakeFS worth using

Now comes the part that justifies the whole tool. Suppose you want to fix a data problem, say the score for Bassem was entered wrongly, and you want to try the fix without any risk to main. You create a branch:

lakectl branch create lakefs://my-repo/fix-scores -s lakefs://my-repo/main

The -s flag names the source: the ref the new branch starts from. The command returns instantly, however big the repository is, because it copies nothing. Confirm the branches:

lakectl branch list lakefs://my-repo
lakectl branch show lakefs://my-repo/fix-scores

Both branches now point at the same commit. Edit the file locally, then upload it to the branch, not to main:

printf 'id,name,score\n1,Amal,71\n2,Bassem,74\n3,Carla,88\n' > scores.csv
lakectl fs upload lakefs://my-repo/fix-scores/data/scores.csv -s ./scores.csv

Read the file from both places and notice they differ. This is the magic moment:

lakectl fs cat lakefs://my-repo/main/data/scores.csv
lakectl fs cat lakefs://my-repo/fix-scores/data/scores.csv

main still says 64 and fix-scores says 74. Anyone reading main, a dashboard or a training job, is completely unaffected. Now look at exactly what changed on the branch, commit it, and compare the branch with main:

lakectl diff lakefs://my-repo/fix-scores
lakectl commit lakefs://my-repo/fix-scores -m "Correct Bassem's score"
lakectl diff lakefs://my-repo/main lakefs://my-repo/fix-scores

The first diff shows the uncommitted change. The second, with two refs, compares two branches. By default lakeFS shows the changes the second ref introduced compared with the merge base. Adding --two-way compares the two heads directly, and --prefix data/ narrows it to a folder.

Merging, or throwing it away

If you are happy with the change, merge the branch into main. The source comes first and the destination second:

lakectl merge lakefs://my-repo/fix-scores lakefs://my-repo/main -m "Merge the score fix"

The merge is atomic: readers of main see either the old data or the new data, never a mixture. It creates a merge commit on main. You can check the history again with lakectl log lakefs://my-repo/main.

If you are not happy, you simply do not merge. Delete the branch and nothing else in the repository is affected:

lakectl branch delete lakefs://my-repo/fix-scores

The command asks you to confirm, and -y skips the question in scripts. After the merge you would delete the branch as well, because it has done its job. Deleting a branch deletes the pointer, not the commits that were already merged.

Everyday commands, grouped by what you want to do

By now you have met most of the everyday commands. This section groups them by intent so that you can find the right one quickly, and adds the ones you have not seen. All of them take lakeFS URIs, and all of them print help with --help.

Working with files

Uploading a single file uses -s. To upload a whole folder add -r (recursive), and to read from standard input use -s -:

lakectl fs upload -r lakefs://my-repo/feature/images/ -s ./images
echo "hello" | lakectl fs upload lakefs://my-repo/feature/notes/hello.txt -s -
lakectl fs ls -r lakefs://my-repo/feature/images/
lakectl fs download -r lakefs://my-repo/main/data/ ./out
lakectl fs rm lakefs://my-repo/feature/notes/hello.txt
lakectl fs rm -r lakefs://my-repo/feature/images/

Uploads and downloads go straight between your machine and the object store using pre-signed URLs, which are temporary addresses that lakeFS creates for you. The data does not flow through the lakeFS server, which is why it scales well. Deleting (fs rm) only removes an object from that branch's current state. Older commits still contain it, and that is a feature.

Seeing what changed

lakectl diff lakefs://my-repo/feature
lakectl diff lakefs://my-repo/main lakefs://my-repo/feature
lakectl log lakefs://my-repo/main --amount 5
lakectl log lakefs://my-repo/main --first-parent --no-merges
lakectl show commit lakefs://my-repo/<commit-id>

log accepts filters. --amount limits how many commits to print, --since takes an RFC 3339 timestamp such as 2026-09-01T00:00:00Z, and --prefixes data/,reports/ shows only commits that touched those folders. lakectl show commit prints the details of one commit, including the metadata you attached.

Naming a version

Commits accept key-value metadata that travels with the snapshot, which is useful for recording what produced it:

lakectl commit lakefs://my-repo/main -m "Nightly load" --meta job=nightly --meta source=crm
lakectl tag create lakefs://my-repo/v1.0 lakefs://my-repo/main
lakectl tag list lakefs://my-repo
lakectl fs cat lakefs://my-repo/v1.0/data/scores.csv

The last command reads the file through the tag, so you get the data exactly as it was when you tagged it, no matter how main has moved since. To replace an existing tag you need -f, and to delete one you use lakectl tag delete lakefs://my-repo/v1.0, with --yes to skip confirmation. Because tags are meant to be stable, treat moving one as a big decision.

Undoing things

There are two very different kinds of "undo", and confusing them is a classic beginner trap. Reset throws away uncommitted changes. Revert creates a new commit that undoes an earlier commit.

lakectl branch reset lakefs://my-repo/feature
lakectl branch reset lakefs://my-repo/feature --prefix data/
lakectl branch reset lakefs://my-repo/feature --object data/scores.csv
lakectl branch revert lakefs://my-repo/main <commit-id>

The first command discards every uncommitted change on the branch, the next two limit it to a folder or a single object, and all of them ask for confirmation unless you add -y. The revert command leaves history intact and adds a commit that cancels the target commit's changes. If the commit you want to revert is a merge commit, you must say which parent to keep, using -m 1; without it you get must specify 1-based parent number for reverting merge commit. Both undo operations need a clean branch, and if the branch has uncommitted changes you get an error about an uncommitted changes (dirty branch). Commit or reset first.

Using lakeFS from Python and from S3 tools

Most data work is done in code rather than in a terminal, and lakeFS offers two ways in that do not involve lakectl at all. The first is the Python SDK, and the second is the S3 gateway, which lets almost any tool that can talk to S3 talk to lakeFS unchanged.

The high-level Python SDK

Install the package, called lakefs (the version checked here is 0.16.0, which needs Python 3.10 or newer). It automatically reads the same ~/.lakectl.yaml or LAKECTL_* environment variables you already set up, so in many cases you need no extra configuration.

pip install lakefs

import lakefs

repo = lakefs.repository("my-repo")
main = repo.branch("main")

# a private branch, created instantly
exp = repo.branch("add-dave").create(source_reference="main")

# write an object on the branch
exp.object("data/extra.csv").upload(data=b"id,name,score\n4,Dave,55\n")

# read it back
with exp.object("data/extra.csv").reader(mode="r") as f:
    print(f.read())

# list uncommitted changes, commit, merge
for change in exp.uncommitted():
    print(change)

exp.commit(message="Add Dave", metadata={"source": "manual"})
exp.merge_into(main)

repo.tag("v1.1").create(source_ref="main")

The shape is the same as in lakectl: you get a repository, pick a branch, create objects, commit, merge, tag. If you need to point at a server explicitly, for example in a script that runs somewhere without a config file, build a client:

import lakefs
from lakefs.client import Client

clt = Client(host="http://localhost:8000",
             username="AKIAIOSFOLQUICKSTART",
             password="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY")
repo = lakefs.repository("my-repo", client=clt)

Errors come from lakefs.exceptions. The ones you will meet first are NotFoundException (the repository, branch or object does not exist), ConflictException (a merge conflict or a name that already exists), and ForbiddenException (your credentials are not allowed). Wrapping the merge in a try block that catches ConflictException is a good habit.

Transactions

One feature worth learning early is the transaction. It creates a hidden temporary branch, lets you make several changes, commits them, and merges them back atomically, cleaning up afterwards. If anything inside raises an error, everything is rolled back and main is untouched:

with repo.branch("main").transact(commit_message="Load daily files") as tx:
    tx.object("data/day1.csv").upload(data=b"a,b\n1,2\n")
    tx.object("data/day2.csv").upload(data=b"a,b\n3,4\n")

That single block is the safe-publishing pattern from the start of this guide in a dozen lines: readers of main see both files or neither.

The S3 gateway

The lakeFS server also exposes an S3-compatible endpoint on the same port. In this view the S3 "bucket" is the repository, and the start of the key is the ref (the branch or tag), so the path is s3://<repo>/<branch>/<path>. That means any tool that can be pointed at a custom S3 endpoint works with lakeFS: the AWS CLI, boto3, many query engines and Spark. For example, with the AWS CLI, using a profile that holds your lakeFS keys:

aws --profile lakefs --endpoint-url http://localhost:8000 s3 ls s3://my-repo/main/

And in boto3:

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://localhost:8000",
    aws_access_key_id="AKIAIOSFOLQUICKSTART",
    aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    region_name="us-east-1",
)
print(s3.list_objects_v2(Bucket="my-repo", Prefix="main/data/"))

Two things to remember. Use path-style addressing (the bucket in the path, not in the host name), and use the region us-east-1, which is the gateway's default. If an S3 client reports SignatureDoesNotMatch, suspect the region, the addressing style, or a computer clock that has drifted, since the gateway rejects requests whose timestamps are too far from its own.

The gateway is the path of least resistance for existing tools. The native clients (lakectl and the Python SDK) are more efficient at scale, because they send only metadata through lakeFS and move the bytes directly to and from the object store. For a beginner the difference is invisible, but it is worth knowing that there are two data paths.

Protecting branches and checking data with hooks

Once several people or pipelines write to the same repository, you want rules. lakeFS gives you two simple, beginner-friendly ones. The first is branch protection, which stops anybody from writing directly to an important branch. The second is hooks, which run automatic checks when something happens, such as a commit or a merge.

Branch protection

A protection rule is a pattern. Any branch that matches it rejects direct writes, deletes, commits and resets, so the only way to change it is to merge another branch into it:

lakectl branch-protect add lakefs://my-repo 'main'
lakectl branch-protect list lakefs://my-repo

After adding the rule, try to upload to main directly. You get cannot write to protected branch. That is the rule working as intended. The right response is the workflow you already know: branch, change, commit, merge. If you ever need to remove the rule, lakectl branch-protect delete lakefs://my-repo 'main' does it.

Hooks: automatic checks

A hook is a small piece of automation that lakeFS runs when an event happens. You describe hooks in a YAML action file stored in the repository under the folder _lakefs_actions/. Each action file names the events it reacts to, and the ordered list of hooks to run. A hook can be a webhook (lakeFS calls a URL you run), a Lua script (run inside lakeFS), or an Airflow DAG trigger. The events that matter first are pre-commit and pre-merge. A pre- hook that fails aborts the operation, and that is the mechanism behind "validate before you publish". A post- hook that fails does not undo anything.

Here is a pre-merge action that calls a webhook on your own validation service before anything is merged into main:

name: validate before merge
on:
  pre-merge:
    branches:
      - main
hooks:
  - id: quality_check
    type: webhook
    description: Ask the validation service to approve this merge
    properties:
      url: "http://validator.internal:8080/check"
      timeout: 1m

Because the action file lives in the repository, it is versioned like data. You can check a file's syntax before committing it:

lakectl actions validate ./_lakefs_actions/validate.yaml

Action files are otherwise only read when an event fires, so a typo in one might surprise you later, which is why validating first is a good habit. After a run you can see what happened:

lakectl actions runs list lakefs://my-repo
lakectl actions runs describe lakefs://my-repo <run-id>

If a merge is rejected, the error tells you which hook failed, and the hook's log is stored in the repository under _lakefs/actions/log/. Hooks that need a secret, such as an API token, can read it from a server environment variable with the {{ ENV.NAME }} syntax in the webhook's headers or query parameters. The server only exposes variables that start with the configured prefix, which defaults to LAKEFSACTION_. This is a topic for the next level, but it is good to know the pattern exists.

Together, branch protection and a pre-merge hook give you a real data quality gate: nobody writes to main, and nothing gets merged unless a check passed.

Configuration: what you set and where

So far you have run a server that configured itself. The moment you use a real bucket and a real database, you will write a configuration. You do not need every key. A small handful does the job, and lakeFS ships sensible defaults for the rest.

Configuration can come from a YAML file, from environment variables, or both. If you do not pass --config (or -c), the server looks for config.yaml in the current folder, then $HOME/lakefs/config.yaml, then /etc/lakefs/config.yaml, and finally $HOME/.lakefs.yaml. Any key can also be an environment variable: LAKEFS_ plus the key in upper case with dots turned into underscores. A minimal production-style file has just three blocks, the database that holds mutable metadata, the secret key, and the blockstore:

database:
  type: postgres
  postgres:
    connection_string: "postgres://lakefs:secret@db.example.com:5432/lakefs?sslmode=require"
auth:
  encrypt:
    secret_key: "a-long-random-string-you-keep-forever"
blockstore:
  type: s3
  s3:
    region: me-south-1

Run it with lakefs --config config.yaml run. A few notes on what each block means.

The database is called the KV store, and it holds the things that change often: where each branch points, which files are staged but not committed, and the credentials. The options are PostgreSQL, DynamoDB, Azure CosmosDB, or local, an embedded database for a single machine. PostgreSQL is the usual choice, and version 11 or newer is required. The object store holds the data and the immutable commit metadata. These two together are all the state lakeFS has, which is what allows you to run several identical lakeFS servers behind a load balancer.

The secret key under auth.encrypt.secret_key is required, and it is used to encrypt stored credentials. If you lose it or change it, the stored credentials stop working. Keep it in a secret manager, pass it as LAKEFS_AUTH_ENCRYPT_SECRET_KEY, and never commit it to Git. The development default is used only by the local and quickstart modes.

The blockstore is the place your data lives. Its type is one of s3, gs (Google Cloud Storage), azure, or local for learning. For S3 the key setting is the region; if you are working in the Middle East, for example in AWS region me-south-1 (Bahrain) or me-central-1 (UAE), set it here to keep your data in the region your employer's data-residency rules require. lakeFS uses the AWS credential chain, so an instance role or a Kubernetes service account is better than putting keys in the file. For S3-compatible stores such as MinIO, you add endpoint and usually force_path_style: true.

There is one rule worth memorising because it causes a confusing error. A repository remembers the type of blockstore it was created on. If you create repositories on a local blockstore and later restart the server with blockstore.type: s3, the server reports Mismatched adapter detected and refuses to serve those repositories. Switch back rather than forcing it.

You can also deploy lakeFS on Kubernetes with the official Helm chart:

helm repo add lakefs https://charts.lakefs.io
helm install my-lakefs lakefs/lakefs -f conf-values.yaml

The chart's defaults are for development only: a local database and local storage. Real deployments supply the PostgreSQL connection string and the encryption key through the chart's secret values. Helm itself is covered in the Helm guide, and the cloud side of the setup is where a tool such as Terraform is usually used.

Common errors and how to read them

lakeFS errors are usually clear once you know the pattern: the HTTP status says the category, and the message says the specific cause. Here are the ones beginners meet most, with what each really means.

repository not found, branch not found, commit not found, tag not found (404). Nine times out of ten this is a typo, or the old URI syntax with an @. Check that the URI has the form lakefs://repo/branch/path, and list what actually exists with lakectl repo list and lakectl branch list.

failed to create repository: storage namespace already in use (400). Another repository, or the leftover _lakefs data of a repository you deleted, occupies that location. lakeFS deliberately does not delete your data when you delete a repository. Pick a new prefix.

failed to create repository: failed to access storage (400). lakeFS could not write to and read from the namespace. The cause is almost always permissions, the wrong region, or a typo in the bucket name. The identity lakeFS runs as needs permission to read, write and list on that bucket prefix, and the server log has a reason field with the details. Fix the bucket policy or IAM role, then retry.

An invalid_namespace message ending with must match: <blockstore type>. The scheme of your namespace does not match the server's blockstore, for example a gs:// address on an S3 server. Use s3://, gs://, an Azure https://...blob.core.windows.net/... address, or local:// to match.

cannot write to protected branch. A protection rule applies. Do the work on another branch and merge.

conflict found (409). Two branches changed or deleted the same object differently. Redo the change from the current main, or choose a merge strategy deliberately.

commit with no message without specifying the "--allow-empty-message" flag. You ran lakectl commit without -m. Add a message.

uncommitted changes (dirty branch). The operation needs a clean branch. Commit your work or reset it.

unsupported protocol scheme "" from lakectl. No endpoint is configured. Run lakectl config.

HTTP 410 Gone when reading an object. The object was physically removed by garbage collection while an old commit still pointed to it. That is expected behaviour after a cleanup, not a bug. It is covered at the next level.

A server that refuses to start after an upgrade, with lakeFS has no administrator of its own. This is new in 1.87. If you upgraded an installation that already holds repositories but does not have a built-in administrator of its own (for example because its users lived in an external authorization service), the server stops and tells you the fix. Stop the server and run lakefs superuser --user-name admin. Add --access-key-id and --secret-access-key with your existing values if you want to keep the keys your clients already use, or omit them to get new ones. Then start the server again. For a fresh installation you will not see this.

Putting it all together: a safe nightly update

Here is one small project that uses everything above. The story: a table of exam scores in data/scores.csv is published on main and read by a dashboard. Each night you need to add new rows, check them, and publish them, and you want a guarantee that a bad file never reaches the dashboard. You will also keep a tagged copy of the monthly version.

First, set up the repository with protection and a first version. Assume the server is running and lakectl is configured:

lakectl repo create lakefs://exam-results local://exam-results
printf 'id,name,score\n1,Amal,71\n2,Bassem,64\n3,Carla,88\n' > scores.csv
lakectl fs upload lakefs://exam-results/main/data/scores.csv -s ./scores.csv
lakectl commit lakefs://exam-results/main -m "Initial scores" --meta job=setup
lakectl tag create lakefs://exam-results/v2026-09 lakefs://exam-results/main
lakectl branch-protect add lakefs://exam-results 'main'

The tag v2026-09 freezes the September state. Protection now means nobody can write to main directly. Next, the nightly job is a short Python script. It works on a branch named after the date, validates the new data, and merges only if the validation passes:

import csv
import io
import datetime
import lakefs
from lakefs.exceptions import ConflictException

repo = lakefs.repository("exam-results")
main = repo.branch("main")
name = "nightly-" + datetime.date.today().isoformat()
branch = repo.branch(name).create(source_reference="main")

# 1. read the current file, add tonight's rows
with main.object("data/scores.csv").reader(mode="r") as f:
    rows = list(csv.DictReader(f))
rows.append({"id": "4", "name": "Dave", "score": "55"})

# 2. validate before publishing
for row in rows:
    if not (0 <= int(row["score"]) <= 100):
        branch.delete()
        raise SystemExit("bad score found; branch discarded, main untouched")

# 3. write the new version to the branch
out = io.StringIO()
writer = csv.DictWriter(out, fieldnames=["id", "name", "score"])
writer.writeheader()
writer.writerows(rows)
branch.object("data/scores.csv").upload(data=out.getvalue())

# 4. commit with metadata, then merge atomically
branch.commit(message="Nightly load " + name, metadata={"job": "nightly"})
try:
    branch.merge_into(main)
except ConflictException:
    raise SystemExit("conflict: someone else changed scores.csv; investigate")
branch.delete()
print("published", name)

Run it with python nightly.py. Then look at the results with the commands you know:

lakectl log lakefs://exam-results/main --amount 3
lakectl fs cat lakefs://exam-results/main/data/scores.csv
lakectl fs cat lakefs://exam-results/v2026-09/data/scores.csv

The first shows the new merge commit. The second shows four rows on main. The third reads the tag and shows the original three rows, proving that the September version is intact and readable. If you change the script so the new score is 150, the validation fails, the branch is deleted, and main never changes: the dashboard never sees a bad file. That is the entire idea of lakeFS in one script. If a bad change ever does get merged, you can put things right with lakectl branch revert lakefs://exam-results/main <commit-id>, but because the merge was a merge commit you would add -m 1.

What you can now do, and what comes next

You can now explain why data needs version control, and you can name lakeFS's core nouns: repository, storage namespace, branch, commit, tag, ref, and merge. You can install and start a server in several ways and verify it. You can configure lakectl, create a repository, upload, inspect, commit, branch, diff, merge, tag, reset and revert, and you know the difference between the last two. You can do the same from Python, including transactions, and through the S3 gateway with standard tools. You can protect a branch, sketch a validation hook, read a configuration file, and read the common error messages. You also know what changed in 1.87: the licence is the Business Source License, and Community has a single administrator.

Mid-level takes these topics further. It covers importing existing data into lakeFS without copying it, how commits and merges work under the hood (ranges and the metarange), cherry-picking, richer hooks with Lua and Airflow, working on a local copy with lakectl local, Spark and Delta Lake integration, garbage collection and retention rules, and using lakeFS with pipelines in CI.

Senior covers running lakeFS for a team: the architecture and its failure modes, sizing and the key-value store, security and the trust model, upgrades and migrations, backup and disaster recovery, cost control, and when the commercial editions or another tool are the better choice.

Where to go next in this catalogue: lakeFS is often used alongside DVC, which takes a different approach to versioning data; with MLflow to tie a model run to a data commit; and with Delta Lake for table-level versioning inside a lakeFS repository.

Sources

• lakeFS documentation home: https://docs.lakefs.io/
• Editions: https://docs.lakefs.io/editions/
• Quickstart: https://docs.lakefs.io/quickstart/launch/
• Architecture concepts: https://docs.lakefs.io/concepts/architecture/
• Glossary: https://docs.lakefs.io/resources/glossary/
• Installation: https://docs.lakefs.io/admin/install/
• Sizing guide: https://docs.lakefs.io/admin/sizing-guide/
• Protecting branches: https://docs.lakefs.io/guides/protect-branches/
• Hooks: https://docs.lakefs.io/guides/hooks/
• lakectl command reference: https://docs.lakefs.io/reference/cli/oss/
• Configuration reference: https://docs.lakefs.io/reference/configuration/
• S3 gateway reference: https://docs.lakefs.io/reference/s3/
• Python SDK: https://docs.lakefs.io/reference/python/getting-started/
• Release notes for v1.87.0: https://github.com/treeverse/lakeFS/releases/tag/v1.87.0
• Licence file (Business Source License 1.1): https://github.com/treeverse/lakeFS/blob/v1.87.0/LICENSE
• Helm chart: https://github.com/treeverse/charts/tree/master/charts/lakefs

This is the interview companion to the beginner guide. It assumes you have read the beginner guide or used lakeFS for a few hours. Interviewers for data and MLOps roles rarely expect deep lakeFS internals from a junior candidate. They want to hear that you understand the problem, use the right words, can run the basic commands without looking them up, and know the traps. Everything below is checked against lakeFS 1.87.

The thirty-second answer

lakeFS is a version-control layer for data that sits on top of an object store such as S3, GCS or Azure Blob. It gives you Git-like repositories, branches, commits, tags and merges over the files in your bucket, without copying the data: branching is a metadata-only operation that takes roughly constant time.

The reason people use it is safety and reproducibility. You change data on an isolated branch, validate it, and merge it atomically so readers never see a half-finished state. If something goes wrong you revert, and if you need last quarter's dataset you read it from a commit or a tag. The data stays in your bucket and in your formats, and tools reach it through the lakectl command line, the Python SDK, or an S3-compatible gateway.

If you have time for one extra sentence, add the current-version fact that shows you are up to date: since version 1.87 the free Community edition is under the Business Source License and has a single administrator user, while multiple users and fine-grained permissions belong to the commercial editions.

The model you must be able to draw

Interviewers like a whiteboard. If asked to explain lakeFS, draw three boxes and name the nouns as you go.

A repository is a namespace of objects with its branches, commits and tags, and it has a storage namespace, the bucket prefix where its data is written. A branch is a pointer to a commit plus uncommitted changes. A commit is an immutable snapshot of the entire repository. A tag is an immutable name for one commit. A ref is any of those three. A merge combines a source ref into a destination branch, detecting conflicts per whole object. The URI shape is lakefs://<repo>/<ref>/<path>.

The sentence that shows understanding is the answer to "why is branching cheap?": because a branch is a pointer, and objects written by lakeFS are never modified, so a new branch refers to the same physical objects as its parent. Only the changes made afterwards cost storage.

A slightly deeper answer, if asked where the state lives: committed metadata is stored as immutable files under _lakefs/ inside the storage namespace, and the things that change often, such as where each branch points and what is staged but not committed, live in a key-value database, typically PostgreSQL. The object store and that database are the only state, so the lakeFS server itself is stateless and you can run several copies behind a load balancer.

Commands to have ready

You should be able to write these without hesitation. Practise them until the argument order is automatic, because the most common live-coding mistake is swapping source and destination.

lakectl repo create lakefs://my-repo s3://my-bucket/my-repo
lakectl branch create lakefs://my-repo/feature -s lakefs://my-repo/main
lakectl fs upload lakefs://my-repo/feature/data/x.csv -s ./x.csv
lakectl diff lakefs://my-repo/feature
lakectl commit lakefs://my-repo/feature -m "add x"
lakectl merge lakefs://my-repo/feature lakefs://my-repo/main
lakectl tag create lakefs://my-repo/v1.0 lakefs://my-repo/main
lakectl log lakefs://my-repo/main
lakectl branch revert lakefs://my-repo/main <commit-id>
lakectl branch reset lakefs://my-repo/feature

Notes to say out loud while you type. The -s on branch create names the source of the new branch. For merge, the source comes first and the destination second, and the order matters. lakectl commit needs a message, or it refuses. reset discards uncommitted work, and revert makes a new commit that undoes an old one; if the target is a merge commit you add -m 1 to choose the parent.

For the connection setup, the interviewer may ask how lakectl knows where the server is. The answer is lakectl config, which writes ~/.lakectl.yaml with the access key ID, the secret and the server endpoint, or the environment variables LAKECTL_CREDENTIALS_ACCESS_KEY_ID, LAKECTL_CREDENTIALS_SECRET_ACCESS_KEY and LAKECTL_SERVER_ENDPOINT_URL. To verify, lakectl identity and lakectl doctor, and curl on /_health for the server.

From Python, the high-level package is lakefs:

import lakefs

repo = lakefs.repository("my-repo")
branch = repo.branch("feature").create(source_reference="main")
branch.object("data/x.csv").upload(data=b"id,v\n1,2\n")
branch.commit(message="add x")
branch.merge_into(repo.branch("main"))

The classic traps and why they happen

Interviewers ask about traps because they reveal whether you have used the tool. Here are the ones to know, each with the reason behind it.

Writing to main directly. A beginner uploads to main because it is the default, and readers instantly see a half-finished change. The fix is a workflow habit (always branch) and a server-side rule: lakectl branch-protect add lakefs://my-repo 'main'. Afterwards a direct write fails with cannot write to protected branch.

Confusing reset with revert. Reset throws away uncommitted changes and cannot be undone, because nothing recorded them. Revert adds a new commit that cancels an earlier one and keeps history. Say this sentence in an interview and you will be remembered.

Expecting file-level merging. lakeFS detects conflicts per object. If two branches both changed scores.csv, you get conflict found (HTTP 409) even if they edited different lines. The answer is to redo the change from the current main, or to pick --strategy source-wins or --strategy dest-wins, knowing that the choice applies to every conflicting object in that merge.

Thinking deletes free the space. Deleting a file on a branch, or deleting a branch, does not remove bytes from the bucket, because old commits still refer to them. Space is reclaimed by garbage collection, a separate job you configure with retention rules. After collection, reading an object that an old commit still references gives HTTP 410 Gone. This is a mid-level topic, but naming it shows awareness.

Forgetting the endpoint. Since 1.74.1 lakectl has no default server address. Without one, requests fail with unsupported protocol scheme "". Always set it with lakectl config or LAKECTL_SERVER_ENDPOINT_URL.

Changing the blockstore type. A repository remembers the type of storage it was created on. Restarting the server with a different blockstore.type gives Mismatched adapter detected. The right response is to revert the setting rather than forcing the server with the override flag.

Reusing a storage namespace. Two repositories cannot share a namespace, and lakeFS does not delete data when you delete a repository, so recreating my-repo on the same prefix fails with storage namespace already in use.

Trusting old tutorials on users and permissions. On Community 1.87 there is one administrator with one credential pair. Commands for groups and policies return "not implemented". If an interviewer asks how you would give analysts read-only access, the honest answer is: that is a commercial-edition capability (role-based access control), and on Community everyone shares the admin key, so I would not hand that key to people who should not have full access.

Forgetting what the quickstart is. Quickstart mode has fixed, publicly documented credentials, local storage and a local database. It is for learning, never for real data.

Common interview questions

Q: What problem does lakeFS solve? A: Data in an object store is a single mutable thing. Overwrites lose the old state, readers see changes instantly, and there is no consistent snapshot across many files. lakeFS adds Git-like branches, commits and merges over the object store without copying data, so I can test changes in isolation, publish them atomically, and go back to an exact earlier version.

Q: How is that different from S3 object versioning? A: Bucket versioning keeps old versions of individual objects. It cannot give me a consistent view of a set of files at one moment, and it has no branches or atomic multi-file publish. lakeFS versions a whole repository as one snapshot and adds isolation and merge.

Q: How is it different from Delta Lake or Iceberg? A: Those are table formats with their own transaction logs and time travel, and they version one table at a time. lakeFS is format-agnostic and versions the whole repository: raw files, several tables, images and model artifacts in one commit. They work together, and lakeFS can even export Delta metadata. See the Delta Lake guide.

Q: How is it different from DVC? A: Both version data for reproducibility. DVC stores small pointer files in Git and pushes data to remote storage. lakeFS runs as a server in front of the object store and gives branches and atomic merges on the data itself, which suits shared, continuously written data lakes. See the DVC guide for the other approach.

Q: Why is creating a branch instant? A: A branch is a pointer to a commit, and committed data is immutable. Creating one copies no objects. The documentation puts branch operations at under about 30 milliseconds at the 90th percentile regardless of repository size.

Q: What is the storage namespace? A: The location in the object store, such as s3://bucket/prefix, where a repository's data and metadata are written. It is assigned at creation, it must match the server's blockstore type, and two repositories cannot share one.

Q: Walk me through making a safe change to production data. A: Create a branch from main, write the new data on the branch, run validation, commit with a message and metadata, merge into main. The merge is atomic, so readers see the old or the new state, never a mix. If the check fails I delete the branch and main is untouched. I would protect main so it only accepts merges, and run the validation as a pre-merge hook so it cannot be skipped.

Q: What is a hook? A: An automated action defined in a YAML file in the repository's _lakefs_actions/ folder. It runs on events such as pre-commit or pre-merge. The types are webhook, Lua and Airflow. If a pre- hook fails, the operation is aborted, which gives me a data quality gate. A failing post- hook does not roll anything back.

Q: How do existing tools read data from lakeFS? A: Through the S3-compatible gateway. The bucket is the repository and the key starts with the ref, so s3://my-repo/main/data/. I point the tool at the lakeFS endpoint, use path-style addressing, and use the lakeFS keys. The native clients move bytes directly to the object store with pre-signed URLs and are more efficient at scale.

Q: What does a commit contain? A: An immutable snapshot of the repository, a committer, a timestamp, a message, and optional key-value metadata, identified by a content-based commit ID. I use the metadata to record the job name or source so I can trace a snapshot back to what produced it.

Q: What would you do if a bad change was merged to main? A: Revert it with lakectl branch revert lakefs://repo/main <commit-id>, adding -m 1 if it was a merge commit. That creates a new commit undoing the change, leaves history intact, and takes moments, since it is a metadata operation.

Q: What is the licensing situation? A: From 1.87 the Community edition uses the Business Source License 1.1. Production use of the unmodified software is allowed for an organisation's own internal business purposes, but not for offering lakeFS to third parties, and each version converts to Apache 2.0 four years after release. Some documentation pages still say Apache 2.0, and those are out of date.

Q: What changed for users and security in 1.87? A: The pluggable IAM and ACL server were removed, so Community runs with a single administrator with one credential pair. Upgrading an installation that had users elsewhere can stop the server with lakeFS has no administrator of its own, and the fix is lakefs superuser --user-name <name>, optionally keeping the existing key pair with --access-key-id and --secret-access-key.

Q: Where does lakeFS keep its state, and why does that matter? A: Data and immutable commit metadata in the object store, and mutable metadata such as branch pointers and staged changes in a key-value database like PostgreSQL. Because the server keeps no state of its own, you can run several behind a load balancer and replace them freely.

Q: Which is riskier, reset or revert? A: Reset, because it permanently discards uncommitted changes that are in no commit. Revert only adds a commit. If in doubt I commit first.

Q: Why might an S3 client fail against the gateway? A: The usual causes are the wrong region (the default is us-east-1), virtual-host addressing instead of path-style, or a clock that has drifted far from the server, since the gateway validates request timestamps.

Q: How would you organise branches for a nightly pipeline? A: One short-lived branch per run, named with the date, created from main. The pipeline writes, validates, commits and merges, then deletes the branch. A failed run leaves main unchanged and the branch can be kept for debugging.

What a strong answer adds

A junior answer lists commands. A strong answer connects them to a why: atomic publish, isolation, reproducibility. It uses the precise words, namely ref, namespace, merge base and uncommitted changes, and it mentions what is not versioned by magic. For example, strong candidates volunteer that conflicts are per object, that deletes do not free space until garbage collection, and that Community has one administrator.

They also show honesty about limits. If you are asked about something you have not used, such as garbage collection settings or importing data, say what you know, say you have not run it, and describe how you would find out: the docs, the lakectl help, and a throwaway quickstart server. Interviewers respond much better to that than to a confident wrong answer.

Mention the ecosystem briefly when it fits. lakeFS is commonly combined with MLflow so a run records which data commit it used, with Spark for large processing, and with orchestrators such as Airflow, which can be called from hooks.

Practical scenario prompts

Some interviews give a short scenario instead of asking definitions. Practise answering these in two or three sentences each.

"The dashboard showed wrong numbers for an hour last night." Ask which job wrote to which location. With lakeFS the fix for the present is to revert the commit that introduced the bad data, and the fix for the future is to make the job write to a branch and merge only after a validation hook passes, with main protected.

"A data scientist needs to try a cleaning step on the full dataset without affecting anyone." Create a branch from main. It costs no extra storage until they change files, they work on it with their usual tools through the gateway or the Python SDK, and they delete it or merge it when done.

"We need to retrain last March's model on exactly the same data." If a tag or commit ID was recorded with the model, read the data through that ref, for example lakefs://my-repo/<commit-id>/data/. If nothing was recorded, use lakectl log with --since to find the right commit, then start tagging from now on.

"Two jobs changed the same file and the merge failed." Explain the per-object conflict rule, then decide which version is right, or redo the second job's change on top of the new main. Mention that strategy flags apply to every conflicting object.

Sixty-second self-test

Answer each out loud, then check against this guide.

1. Say in two sentences what lakeFS is and where your data lives.
2. Name the five nouns: repository, storage namespace, branch, commit, tag.
3. Write the URI of data/x.csv on branch feature in repository my-repo.
4. Why is creating a branch instant, and what does it cost in storage?
5. Write the commands to branch, upload, commit and merge, with the merge argument order.
6. What is the difference between reset and revert?
7. What is a per-object conflict, and what are the two merge strategies?
8. What does a failing pre-merge hook do?
9. What two changes did 1.87 make to the licence and to users?
10. Which tool reads data through the S3 gateway, and what three settings must you get right?

If you hesitated on more than two, reread the matching sections of the beginner guide and run the commands on a quickstart server. Ten minutes at a real terminal beats an hour of rereading.

These are the mistakes beginners actually make with lakeFS, each with the symptom you will see, the cause, and the fix. After them come the habits worth building on day one and a short checklist. Everything is checked against lakeFS 1.87. Read the mistakes once now, and come back to them when something goes wrong, because most lakeFS problems are one of these.

Mistakes beginners actually make

Mistake 1: Writing straight to main

Symptom. A dashboard or a model shows strange numbers for a while. The data on main is half old and half new, and nobody can say exactly when it changed.

Cause. main is the default branch, so it is the easiest place to write. A long upload or job writes files one at a time, and anyone reading main in the middle sees a mixture. lakeFS can only protect readers if you give it the chance to.

Fix. Do every change on a branch and merge when it is ready. Make the server enforce it:

lakectl branch-protect add lakefs://my-repo 'main'

After this, a direct write fails with cannot write to protected branch, which is the rule doing its job. Change the habit, not the rule.

Mistake 2: Confusing reset with revert

Symptom. Someone runs lakectl branch reset to "undo the last change" and discovers the work is gone. Or someone runs revert and is puzzled that there is a new commit instead of the old one disappearing.

Cause. The two commands undo different things. Reset discards uncommitted changes, which exist in no commit, so they cannot come back. Revert adds a new commit that cancels an earlier commit and leaves the history alone.

Fix. Learn the rule: uncommitted work gets reset (carefully), committed work gets reverted. When unsure, commit first. A commit you regret is easy to revert, and an uncommitted change you reset is gone forever.

lakectl branch reset lakefs://my-repo/feature --prefix data/
lakectl branch revert lakefs://my-repo/main <commit-id>

Mistake 3: Reverting a merge commit without a parent number

Symptom. must specify 1-based parent number for reverting merge commit.

Cause. A merge commit has two parents: the destination branch first, then the source. lakeFS cannot guess which side you want to go back to. Cherry-picking a merge commit has the same rule.

Fix. Add -m 1 to keep the destination's side, which is what you want when you are undoing "the merge that brought in the bad data".

lakectl branch revert lakefs://my-repo/main <merge-commit-id> -m 1

Mistake 4: Swapping merge source and destination

Symptom. main does not change, and the feature branch suddenly contains everything from main, or the wrong branch receives the result.

Cause. The order is lakectl merge <source> <destination>. People type it in the order they say it in English.

Fix. Read the command as "merge THIS into THAT". Before the first run, look at the two URIs and say aloud which one should change. Afterwards, lakectl log lakefs://my-repo/main shows the merge commit on the branch you intended.

lakectl merge lakefs://my-repo/feature lakefs://my-repo/main -m "Merge feature"

Mistake 5: Expecting lakeFS to merge inside a file

Symptom. conflict found (HTTP 409) even though two people edited different lines of the same CSV.

Cause. lakeFS treats an object as a single unit. If both sides changed the same object differently, or one changed it and the other deleted it, that is a conflict. It never opens a file and combines lines.

Fix. Redo the change on a new branch from the current main, so that it applies on top of the other person's work. Use --strategy source-wins or --strategy dest-wins only when you know one side should win for every conflicting object in that merge, since the flag is not per file. Better, avoid the problem by designing data as many files (one per partition or per day) rather than one giant file that everybody edits.

Mistake 6: Believing deletes free space

Symptom. You delete a large folder and a branch, and the bucket does not shrink. The storage bill keeps climbing.

Cause. Old commits still refer to those objects, and objects written by lakeFS are never modified or removed by ordinary operations. Branching is free because nothing is copied, but it also means nothing is freed until you ask.

Fix. Garbage collection is a separate job with retention rules, covered in the mid-level guide. For now, understand that "delete" in lakeFS means "remove from this branch's current state". Delete branches you no longer need, because an object that is in the head of any branch is never collected, and plan for garbage collection before the data gets big. A related symptom, HTTP 410 Gone when reading an old object, simply means garbage collection removed bytes that an old commit still names. It is expected.

Mistake 7: Forgetting to commit

Symptom. You uploaded files, you can read them on the branch, but a colleague reading a tag or a commit ID cannot see them. Or a merge reports no changes.

Cause. Uploads are uncommitted changes. They are visible on the branch but are not part of history. Merging and tagging work on commits, so uncommitted work is left behind.

Fix. Make lakectl diff lakefs://my-repo/<branch> part of your routine before merging. If it lists changes, commit them. If a commit says no changes, there was nothing staged, and you can safely move on.

lakectl diff lakefs://my-repo/feature
lakectl commit lakefs://my-repo/feature -m "Describe what changed and why"

Mistake 8: Committing with no message, or a useless one

Symptom. commit with no message without specifying the "--allow-empty-message" flag. Or, months later, a log full of "update", "fix" and "test" that nobody can use.

Cause. lakectl refuses an empty message on purpose. People then type the shortest thing that works.

Fix. Write the message you would want to read in six months: what changed and why. Add metadata for machine-readable context, which is where the job name, the source system or the Git commit of your code belongs.

lakectl commit lakefs://my-repo/main -m "Load September CRM export" --meta job=nightly --meta source=crm

Mistake 9: Using the old URI syntax

Symptom. repository not found or branch not found for something you can see in the web page.

Cause. Older blog posts write lakefs://repo@branch/path. That @ separator was replaced with a slash. The current form is lakefs://<repo>/<ref>/<path>.

Fix. Use slashes. Also check for plain typos, and list what exists with lakectl repo list and lakectl branch list lakefs://my-repo.

Mistake 10: Not setting the lakectl endpoint

Symptom. Every lakectl command fails with something like unsupported protocol scheme "", although the server is running fine.

Cause. Since 1.74.1, lakectl no longer assumes http://localhost:8000. The generated command reference still suggests a default, which adds to the confusion.

Fix. Set the endpoint explicitly, either by running lakectl config or with the environment variable:

export LAKECTL_SERVER_ENDPOINT_URL="http://localhost:8000"

Mistake 11: Storage namespace problems

Symptom. Creating a repository fails with storage namespace already in use, or failed to access storage, or an invalid_namespace error.

Cause. These are three different problems. Already in use: another repository, or the _lakefs leftovers of a deleted one, occupies that location, because lakeFS does not delete data when you delete a repository. Failed to access storage: lakeFS cannot read and write there, usually IAM permissions, the wrong bucket name, or the wrong region. Invalid namespace: the scheme does not match the server's blockstore, such as gs:// on an S3 server.

Fix. Choose a fresh prefix for the first, repair the permissions and region for the second (the server log has a reason field), and use the right scheme for the third: s3://, gs://, an Azure https:// address, or local://.

Mistake 12: Changing the blockstore type after creating repositories

Symptom. After a restart, the server logs Mismatched adapter detected. lakeFS started with adapter of type ....

Cause. A repository remembers the type of storage it was created on. If you created it on the local blockstore and restart the server configured for S3, the two no longer match.

Fix. Put the blockstore.type setting back. There is a flag to start anyway, --allow-mismatched-repos, but the behaviour is undefined, and it is meant only for cleaning up. Do not treat it as a solution.

Mistake 13: Losing or changing the encryption secret

Symptom. After redeploying with a new configuration, the stored credentials no longer work and nobody can log in.

Cause. auth.encrypt.secret_key encrypts what lakeFS stores, including credentials. A different key cannot decrypt what the old key encrypted. A random key generated on every start is the usual way this happens in containers.

Fix. Generate one long random string, store it in a secret manager, and inject it through LAKEFS_AUTH_ENCRYPT_SECRET_KEY. Keep it for the life of the installation. If you did lose it on a Community server, you can create a fresh administrator with lakefs superuser --user-name admin.

Mistake 14: Using quickstart for something real

Symptom. It works, then you notice the server refuses production settings, or you realise the credentials in your repository are the public ones from the docs.

Cause. Quickstart mode is a teaching tool: local database, local storage and fixed credentials that everyone knows. Mixing it with real settings gives FATAL: quickstart mode can only run with local settings.

Fix. Use quickstart to learn. For anything that matters, run a real configuration with PostgreSQL (or DynamoDB), a real bucket, and your own administrator credentials.

Mistake 15: Expecting users, groups and permissions on Community

Symptom. A tutorial says to create a group and a policy, but the command returns not implemented, and the UI page does not exist. Or you try to create a second key pair and get exceeded number of allowed credentials.

Cause. Since 1.87, Community has one administrator with one credential pair. Users, groups, policies and single sign-on are commercial-edition features. The tutorial was written for an older release.

Fix. Check the date of any tutorial and compare it with the release notes. Do not hand the admin key to people who should not have full access. If you need per-person permissions, look at the commercial editions rather than trying to simulate them.

Mistake 16: Upgrading without reading the 1.87 notes

Symptom. After upgrading, the server refuses to start with lakeFS has no administrator of its own, or the log fills with "is deprecated. Value is no longer used" warnings for auth.* keys.

Cause. 1.87 removed the pluggable IAM and the ACL reference server. An installation with data but no built-in administrator must create one, and the old auth.api.*, auth.oidc.* and related keys are now ignored.

Fix. Before upgrading a deployment that used an external authorisation service, read the release notes. Stop the server and run lakefs superuser --user-name admin --access-key-id <existing> --secret-access-key <existing> to keep your clients working, then remove the dead keys from the configuration. Also avoid versions 1.48.0 and 1.74.1 when choosing a pin.

Mistake 17: Path-style and region problems with S3 clients

Symptom. An S3 tool pointed at lakeFS fails with SignatureDoesNotMatch or reports an expired request.

Cause. The gateway expects path-style addressing and the region us-east-1 by default, and it validates the timestamp on each request. A client using virtual-host style addressing, a different region, or a laptop whose clock has drifted will be rejected.

Fix. Configure path-style addressing, region us-east-1, the lakeFS endpoint and the lakeFS keys. Synchronise the clock. In Spark's S3A connector the relevant setting is fs.s3a.path.style.access=true.

Mistake 18: Large uploads failing behind a proxy

Symptom. Small files upload fine, but bigger ones fail with HTTP 413 when lakeFS runs on Kubernetes behind NGINX.

Cause. The NGINX ingress default body limit is 1 MiB, and the request is being stopped before it reaches lakeFS.

Fix. Raise the ingress proxy-body-size setting. Remember that the native clients send data directly to the object store with pre-signed URLs, so this mostly affects the S3 gateway and the web upload.

Mistake 19: Sharing the credentials in code

Symptom. An access key and secret appear in a notebook, a script, or a Git repository.

Cause. With one administrator key on Community, every client uses the same credentials, so it is tempting to paste them where they are needed.

Fix. Keep credentials in ~/.lakectl.yaml with restricted file permissions, or in environment variables injected by a secret manager. The Python SDK reads LAKECTL_* variables automatically, so you rarely need to write them in code. If a key has been exposed, rotate it by running lakefs superuser with new values and updating your clients.

Habits worth adopting on day one

Branch for everything. Since branches cost almost nothing, every pipeline run, experiment and manual fix deserves its own short-lived branch named after what it is, such as nightly-2026-10-01 or fix-scores. Delete it after the merge. A repository with hundreds of abandoned branches is hard to reason about, and the objects in any branch head cannot be collected.

Protect main from the start. Adding the rule on the first day is one command, and it turns the rest of this list from advice into enforcement.

Write commit messages and metadata as if a stranger will read them. The stranger is you, in a year, trying to reproduce a result.

Record the commit ID with every result. Whenever you train a model, publish a report or export a dataset, write down the lakeFS commit ID (or a tag) of the input. That single habit is the practical meaning of "reproducible". If you use MLflow, log it as a parameter or tag on the run.

Tag anything you might need again. Branches move, tags do not. Tag the dataset you published, the snapshot you trained on, the state you sent to a customer. lakectl tag create lakefs://my-repo/v2026-10 lakefs://my-repo/main takes a second.

Always diff before you merge. lakectl diff on the branch, and again between the branch and main, shows what you are about to publish. Reading a short list of changes catches most mistakes.

Validate before publishing. Even a five-line check in a script, run on the branch before the merge, is worth more than a clever recovery plan. Later, move the check into a pre-merge hook so nobody can forget it.

Use the quickstart to try risky commands. If you are unsure what reset, revert or a merge strategy will do, try it on a throwaway server with throwaway data. It takes a minute to start, and nothing you do there matters.

Keep lakectl and the server on matching versions. Mismatched versions have caused real failures, including a panic fixed in 1.74.2 when a newer lakectl talked to an older server. Check both with --version.

Learn to read lakectl <command> --help. The argument order and flags are documented right there, and it is the fastest way to confirm a flag without searching the web. Combine it with lakectl doctor when the tool itself misbehaves.

Mind your region. If your employer or your customers have data-residency rules, for example for data that must stay in the Gulf or in Egypt, pick a bucket in an appropriate region such as AWS me-south-1 or me-central-1 and set blockstore.s3.region to match. lakeFS stores data where you tell it to, so the decision is yours and should be made before the first repository is created.

Read the release notes before any upgrade. Especially now, with the 1.87 licence and authentication changes, upgrading blindly is the one habit that reliably hurts.

A short checklist

Use this before you call a first lakeFS setup "done".

• lakefs --version and lakectl --version agree, and print 1.87 or later.
• curl -i http://<host>:8000/_health returns 200.
• lakectl config has been run, the endpoint is set explicitly, and lakectl identity works.
• You are not using quickstart mode for real data, and the encryption secret is stored safely outside the repository.
• The repository's storage namespace uses a scheme that matches the blockstore, and it has its own prefix.
• main is protected, and changes go through branches.
• Every commit has a message, and important snapshots have tags.
• You can explain the difference between reset and revert without looking it up.
• You know that deleting data does not free space until garbage collection runs, and you have put garbage collection on your list of things to learn next.
• Credentials are not written into code, notebooks or Git history.
• You know what the Community edition does and does not include in 1.87, and that it is under the Business Source License 1.1.

A tiny practice session

Spend fifteen minutes on this and you will have touched almost every tip above. Start a quickstart server and configure lakectl. Create a repository called practice-one and put one CSV file into main, then commit and tag it. Protect main, then try to upload another file to it directly, and read the error. Create a branch, change the CSV, and run lakectl diff on the branch before committing. Commit with a real message and two --meta pairs, then diff the branch against main. Merge. Read the file through the tag to see the old version and through main to see the new one. Finally, make a deliberate mistake on a second branch, commit it, merge it, and undo it with branch revert, remembering -m 1 because the target is a merge commit.

When you can do that sequence from memory, and explain what each command did and why, you have the day-one habits. The next step, in the mid-level guide, is importing existing data without copying it, hooks that do real validation, garbage collection, and working with Spark.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. The version pinned throughout is lakeFS 1.87, the release that changed the licence and removed multi-user authentication from the Community edition, so read the security section even if you skip the rest.

Where this picks up

Each section starts with the problem it solves, and most end with a Try it you can run against a dev server in a few minutes.

How lakeFS works under the hood

At Beginner you learned the nouns: repository, branch, commit. To predict performance and failure modes you need the layer below them, because a version-control system over a bucket makes very different promises from a file system.

lakeFS is one stateless Go binary that listens on a single port, 8000 by default, and serves three things at once: the web UI, the OpenAPI server under /api/v1, and an S3-compatible gateway. Inside the binary sit four components: an auth service, the versioning engine called Graveler, a storage adapter for your object store, and the hooks engine. The binary keeps almost no state of its own. State lives in two places, and knowing which piece lives where explains most operational behaviour.

The object store holds two kinds of content. The first is your data: every object you upload lands in the repository's storage namespace under a generated physical address. lakeFS never modifies an object it has written; it only ever adds them, and garbage collection is the one thing that deletes them. The second is committed metadata: each commit is described by immutable SSTable files (the RocksDB-compatible sorted-table format) stored under <namespace>/_lakefs/.

Committed metadata is organised as a two-level Merkle tree. The leaves are ranges, each an SSTable holding a sorted slice of the repository's object paths with their physical addresses. Above them sits a metarange, an SSTable that lists the ranges. A commit points to one metarange. Because ranges are content-addressed and immutable, two commits that share most of their objects share most of their ranges, and nothing is copied.

The KV store holds everything mutable: where each branch points, the staging area of uncommitted changes, credentials, and records of hook runs. PostgreSQL is the usual production choice; DynamoDB and CosmosDB exist for AWS and Azure shops; the local type is an embedded single-node store for development only.

Three consequences follow, and they are worth memorising.

Branching is metadata only. Creating a branch writes one pointer in the KV store. It is constant time whatever the repository size, and the sizing guide puts branch operations under 30 ms at the 90th percentile.

Commit and merge cost scales with the size of the change. A commit folds the staged entries for one branch into new ranges and a new metarange, reusing untouched ranges. A merge of a branch that changed a thousand objects in a billion-object repository does work proportional to the thousand. This is the property that makes lakeFS usable at data-lake scale, and it is why a workflow that commits small and often is cheap.

Uncommitted writes sit in the KV store. The sizing guide estimates about 150 MiB of KV storage per 100,000 uncommitted writes. A job that writes ten million objects and never commits puts real pressure on PostgreSQL. Commit in batches, or make the job commit at the end of each stage.

Two more facts from the architecture matter daily, and both are easy to verify on a dev server once you know to look for them. The local disk is only a cache (committed.local_cache, 1 GiB by default), so losing a replica loses nothing. And every lakeFS read resolves through the same path: ref to commit, commit to metarange, metarange to range, range to physical address. Reading a branch head additionally merges in that branch's staged entries from the KV store.

Merges, conflicts and undoing things

Beginner merges were clean because one branch changed things. Real pipelines have concurrent writers, so you need the exact rules.

A merge takes a source ref and merges it into a destination branch. lakeFS finds the merge base, the nearest common ancestor, and performs a three-way comparison at the level of whole objects. There is no in-file merge: lakeFS has no idea what is inside a Parquet file. Each path falls into one case:

• Changed only on one side: the change is taken.
• Changed identically on both sides: no conflict.
• Changed differently on both sides, or changed on one side and deleted on the other: a conflict, reported as HTTP 409 conflict found.

The resulting merge commit has two parents, and the order matters: the first parent is the destination, the second is the source. That ordering is what -m 1 refers to when you later revert a merge.

lakectl merge lakefs://my-repo/feature lakefs://my-repo/main -m "Promote feature"
# conflicts? choose a blanket strategy, which applies to every conflict in the merge
lakectl merge lakefs://my-repo/feature lakefs://my-repo/main --strategy source-wins
lakectl merge lakefs://my-repo/feature lakefs://my-repo/main --strategy dest-wins

The strategies are blunt. source-wins and dest-wins resolve all conflicts the same way, so they suit cases where you know one side is authoritative, such as a backfill branch that should override. For everything else, the honest resolution is to fix the content on the feature branch, recommit, and merge again. Because datasets are files, "resolving" usually means re-running the job that produced the conflicting object.

Undoing: reset, revert, cherry-pick

Three operations cover different regrets, and choosing the wrong one is a common mid-level mistake.

lakectl branch reset lakefs://my-repo/feature --prefix tmp/ -y
lakectl branch revert lakefs://my-repo/main <commit-id>
lakectl branch revert lakefs://my-repo/main <merge-commit-id> -m 1
lakectl cherry-pick lakefs://my-repo/<commit-id> lakefs://my-repo/release

Reverting a merge commit requires -m 1 to say which parent's side to keep, and the error must specify 1-based parent number for reverting merge commit is the prompt. Revert and cherry-pick also refuse to run on a branch with staged changes: the error uncommitted changes (dirty branch) means commit or reset first.

Atomic multi-file changes

A dataset is often several files that must change together: data, an index, a manifest. lakeFS commits are atomic per branch, so the pattern is to write to a scratch branch, commit once, and merge. The Python SDK packages that as a transaction, covered below. The important design rule is that readers never read a branch that is being written. Readers pin a commit or a tag, or read main, which only ever changes by merge.

Choosing the data path: S3 gateway or native clients

Every client reaches data one of two ways, and the choice determines where your bandwidth bottleneck sits.

The S3 gateway makes lakeFS look like an S3 endpoint. The bucket is the repository and the key starts with the ref, so s3://my-repo/main/data/x.csv addresses a path on main. Any tool that speaks S3 works without changes: the AWS CLI, boto3, Spark over S3A, Trino, R. The cost is that data flows through the lakeFS server, so its network bandwidth and CPU become the ceiling.

Native clients (lakectl, the Python SDK, the lakeFSFS Hadoop filesystem) split the work. They ask lakeFS for metadata and a pre-signed URL, then read or write the bytes directly against the object store. Each request to lakeFS is about a kilobyte, so one small server can front very large transfers. This works on S3, GCS and Azure.

# S3 gateway with the AWS CLI: point --endpoint-url at lakeFS, bucket = repository
aws --profile lakefs --endpoint-url https://lakefs.example.com s3 ls s3://my-repo/main/

# native path: metadata via lakeFS, bytes direct to the bucket
lakectl fs download -r lakefs://my-repo/main/data/ ./out

Rule of thumb: use native clients wherever one exists for throughput-heavy jobs, and use the gateway for tools that only speak S3 or for small data. The gateway supports a subset of S3. Get, put, head, delete, batch delete, copy, list (delimiter / only) and multipart upload work; server-side encryption parameters, SelectObject, storage classes and object tagging do not. ListParts and ListMultipartUploads work only when the underlying store is AWS S3. If a tool fails with an unsupported-feature error, check this list before debugging anything else.

Gateway authentication uses SigV4, which includes a timestamp. Since 1.74.1 and 1.75 lakeFS validates request expiry, so a client with a skewed clock gets SignatureDoesNotMatch or an expired-request error. Sync clocks first. Two more gateway settings trip people: the signing region defaults to us-east-1 (gateways.s3.region), so clients should use that region even if the bucket sits in me-south-1 or me-central-1; and virtual-host-style addressing needs gateways.s3.domain_name configured, so use path-style addressing (force_path_style, or fs.s3a.path.style.access=true in Hadoop) unless you have set it up.

Pre-signed URL pitfalls. The UI can also use pre-signed URLs, but blockstore.s3.disable_pre_signed_ui defaults to true. If you turn it off, browsers talk to the bucket directly, which requires a CORS policy on the bucket allowing GET, PUT and HEAD from the lakeFS origin and exposing the ETag header (plus the x-ms-* headers on Azure). Without CORS, the symptom is UI previews and uploads failing while lakectl works fine. For S3, pre_signed_expiry defaults to 15 minutes, which matters for very large single uploads on slow links.

Importing existing data without copying it

Your data lake already exists. Copying petabytes into a new namespace is neither affordable nor necessary, because import registers existing objects with lakeFS as pointers and creates a commit, without moving a byte.

lakectl import --from s3://analytics-raw/events/2026/ \
               --to lakefs://my-repo/main/raw/events/ \
               -m "Import 2026 events"

The rules are strict, and each has a reason.

The source must not lie inside any lakeFS storage namespace. Importing lakeFS's own managed prefixes would make two logical systems claim the same objects. Since 1.54 lakeFS rejects this outright. The import overwrites the destination prefix: it is a snapshot of the source listing at that moment, not a merge. lakeFS needs read access to the source prefix, which on S3 means the bucket policy or IAM role must allow listing and reading there.

Imported objects stay where they are, and garbage collection never deletes them, because they sit outside the namespace. That is a safety feature and also a consequence: if you retire the source bucket or lifecycle-expire those objects, lakeFS will still list them and reads will fail with an error from the object store. Treat the source prefix as part of the repository's durability contract after import. Teams commonly import onto a dedicated branch first, verify with a hook or a row count, and then merge into main.

Importing is a one-time snapshot. If upstream producers keep writing to the source bucket, the new files do not appear in lakeFS until you import again. The clean end state is to point producers at lakeFS, through the S3 gateway or native clients, so that new data is written into the namespace from the start.

For the local blockstore (development only), import is disabled by default. Enable it with blockstore.local.import_enabled: true and list permitted paths under allowed_external_prefixes.

Hooks and branch protection as quality gates

Convention does not protect main; mechanism does. lakeFS gives you two: branch protection rules and actions (its hooks system). Together they turn "please validate before merging" into "you cannot merge until validation passes".

A protection rule is a glob pattern. Any branch matching it rejects direct writes, deletes, commits and resets, so the only way content arrives is a merge. Revert remains allowed, deliberately, so you can always undo.

lakectl branch-protect add lakefs://my-repo 'main'
lakectl branch-protect list lakefs://my-repo

Writing to a protected branch fails with cannot write to protected branch or cannot commit to protected branch. Protection on its own only forces merges; it is the pre-merge hook that makes each merge earn its place.

Action files

An action is a YAML file stored in the repository itself, under _lakefs_actions/. Because it is versioned with the data, the rules that apply to a merge are the rules that exist on the source branch at that moment, and changing the rules goes through the same review as changing data.

name: validate-schema
on:
  pre-merge:
    branches:
      - main
hooks:
  - id: check-parquet-schema
    type: lua
    description: Reject merges that change the event schema
    properties:
      script_path: scripts/check_schema.lua
  - id: notify-data-team
    type: webhook
    description: Tell the validation service
    properties:
      url: "https://dq.example.com/validate"
      timeout: 2m
      headers:
        Authorization: "Bearer {{ ENV.DQ_TOKEN }}"

The vocabulary:

• Events are pre-commit and post-commit, pre-merge and post-merge, create and delete of branches and tags (each with pre- and post-), pre- and post-revert, pre- and post-cherry-pick, and the experimental prepare-commit. The branches filter does not apply to tag events.
• Hook types are lua (a script run inside lakeFS), webhook (an HTTP call), and airflow (triggers a DAG).
• A pre- hook that fails aborts the operation. A post- hook that fails does not roll anything back: the merge already happened.

Hooks run synchronously for Lua and webhook types, so a slow validator slows the merge. The webhook timeout defaults to one minute. The Airflow hook returns once the DAG trigger is accepted, which means it tells you the DAG started, not that it passed. If you need a gate, use a synchronous type, or have the DAG call back and have a pre-merge hook check its result.

Two behaviours surprise people. First, action files are validated only when an event fires, not when committed, so a typo in on: sits silently until the merge that needs it. Catch that with lakectl actions validate ./_lakefs_actions/validate_schema.yaml in CI. Second, the action that governs a merge comes from the source branch's content, so a malicious or careless branch could remove its own gate. Protect main and require that changes to _lakefs_actions/ are themselves reviewed; in Community there is no per-user permission to restrict who can edit them, which is part of why you review the pull request.

Secrets and safety switches

Webhook headers and query parameters can reference server environment variables with {{ ENV.NAME }}. Only variables with a configured prefix are visible: actions.env.prefix defaults to LAKEFSACTION_ in the server code, so the variable must be named LAKEFSACTION_DQ_TOKEN and referenced as {{ ENV.DQ_TOKEN }}. One docs page shows a different spelling; the configurable key and its code default are what to trust, and you can confirm with a failing hook's log. Network access from Lua is off by default (actions.lua.net_http_enabled: false), actions.env.enabled: false blocks environment access, and actions.enabled: false turns all hooks off.

Reading a failed run

lakectl actions runs list lakefs://my-repo --branch main
lakectl actions runs describe lakefs://my-repo <run-id>

Each run writes logs to _lakefs/actions/log/<runID>/<hookRunID>.log in the namespace, plus a run.manifest. When a merge is rejected, runs describe tells you which hook failed, and the log says why.

The Python SDK in production code

The lakefs package is the high-level SDK, currently 0.16.0, requiring Python 3.10 or later; it wraps the generated lakefs-sdk client, whose version tracks the server. It reads credentials from ~/.lakectl.yaml or the LAKECTL_* environment variables, so the same configuration serves the CLI and your code. In a container, prefer the environment variables, because there is no home directory worth writing to.

import lakefs

repo = lakefs.repository("my-repo")
main = repo.branch("main")

# read through a pinned ref, so the job is reproducible
ref = repo.ref("v2026-09-01")          # a tag
with ref.object("data/events.csv").reader(mode="r") as f:
    header = f.readline()

Pinning matters more than anything else in the SDK. A training job that reads main reads whatever main is at that instant; a job that reads a tag or a commit ID reads the same bytes next year. Record the commit ID your run used, alongside the model, in your experiment tracker (see MLflow), and you can reproduce the exact inputs later.

Transactions

from lakefs.exceptions import ConflictException

try:
    with repo.branch("main").transact(commit_message="Refresh daily features") as tx:
        tx.object("features/day.parquet").upload(data=day_bytes)
        tx.object("features/_manifest.json").upload(data=manifest_bytes)
except ConflictException:
    # someone else changed main between the branch point and merge
    raise

transact creates an ephemeral branch off the target, performs your operations on it, commits, merges back into the target, and deletes the ephemeral branch. If your block raises, the changes are rolled back: nothing reaches the target. This is the right default for any multi-file update. Note what it does not do: it does not retry a conflict. Decide, per job, whether to retry with a fresh transaction or fail loudly.

Error handling

The exceptions you will handle live in lakefs.exceptions: NotFoundException for missing repositories, branches or commits, ObjectNotFoundException for missing objects, ConflictException for merge conflicts and name collisions, ForbiddenException for auth failures and protected branches, and ServerException for 5xx. Retry only transient conditions. A ForbiddenException on a protected branch will never succeed on retry. For 429, back off; the lakectl client retries four times by default, and you can tune it in .lakectl.yaml under server.retries.

pandas and fsspec

import pandas as pd
from lakefs_spec import LakeFSFileSystem   # pip install lakefs-spec

fs = LakeFSFileSystem()
df = pd.read_parquet("lakefs://my-repo/main/features/day.parquet")

The lakefs-spec package registers the lakefs:// scheme with fsspec, so pandas, Dask and others read lakeFS paths as if they were files, and it uses pre-signed URLs underneath. That gives you native-path performance with existing code.

Local work and pairing data with code

Notebooks and tools that want a local directory are awkward with a remote store. lakectl local syncs a lakeFS path with a folder, in the way a Git working tree tracks a remote.

lakectl local clone lakefs://my-repo/main/dataset/ ./dataset
cd dataset
lakectl local status .            # what changed locally
lakectl local pull .              # bring remote changes down
lakectl local commit . -m "Fix labels"

The link between the directory and the remote is stored in a .lakefs_ref.yaml file in that directory. By default clone adds a .gitignore so the data itself stays out of Git (--gitignore defaults to true), while the small ref file can be committed. That is the point: commit .lakefs_ref.yaml to your code repository, and each Git commit records exactly which data commit it was written against. Your code and your data then carry matching version pointers, and lakectl local checkout . -r <ref> moves the directory to that data state.

Two limits to know. local commit refuses when the remote has moved since your last pull; pull first, or use --force knowingly. And symlinks and non-regular files are skipped unless configured (local.symlink_support, local.skip_non_regular_files). Since 1.82, local commit accepts -y or --yes to skip the confirmation prompt, useful in scripts.

The enterprise mount feature (Everest) provides a virtual filesystem instead of a sync; it is not part of Community.

Spark, Delta and the compute layer

Most lakeFS volume comes through Spark, and the integration choice is the gateway or lakeFSFS, the same split as before.

Through the S3 gateway you change two settings and keep your s3a:// URLs:

spark.hadoop.fs.s3a.endpoint=https://lakefs.example.com
spark.hadoop.fs.s3a.path.style.access=true
spark.hadoop.fs.s3a.access.key=<lakefs-access-key>
spark.hadoop.fs.s3a.secret.key=<lakefs-secret-key>

The paths become s3a://my-repo/main/events/. Simple, and every byte goes through lakeFS. There is a per-bucket variant, fs.s3a.bucket.<repo>.endpoint, which lets one Spark session talk to both lakeFS and a normal bucket, which you will want as soon as any job mixes them.

Through lakeFSFS (io.lakefs:hadoop-lakefs-assembly, currently 0.18.0) you use lakefs:// paths and get metadata-only traffic to lakeFS with direct data access. You supply both sets of credentials: the lakeFS ones for metadata and S3 ones for the bytes.

spark-submit \
  --packages io.lakefs:hadoop-lakefs-assembly:0.18.0 \
  --conf spark.hadoop.fs.lakefs.impl=io.lakefs.LakeFSFileSystem \
  --conf spark.hadoop.fs.lakefs.endpoint=https://lakefs.example.com/api/v1 \
  --conf spark.hadoop.fs.lakefs.access.key=<lakefs-access-key> \
  --conf spark.hadoop.fs.lakefs.secret.key=<lakefs-secret-key> \
  job.py

Note that here the endpoint includes /api/v1. lakectl appends the API path for you if you omit it; Hadoop does not. The lakeFSFS retry settings (fs.lakefs.api.retry.max-retries, default 5, and backoff keys) are worth raising if a long job sees sporadic 429s.

Isolation patterns that pay off

The winning pattern for batch pipelines is branch per run. The orchestrator (Airflow is the usual one, and lakeFS has an Airflow hook type) creates a branch from main, the job writes to it, validations run against it, and a successful run merges it. A failed run leaves a branch you can inspect, then delete. Consumers reading main never observe a half-written dataset, which removes the class of bugs where a dashboard shows a partial day.

Delta Lake

Delta Lake tables work on lakeFS because Delta is just files. The transaction log lives alongside the data, so branching a repository branches the table. Two cautions. Delta's own concurrency control expects one writer's commits to be atomic on a path, so do not have two jobs write the same table on the same branch, and instead give each its own branch and let lakeFS do the merge. And a merge of two branches that both appended to the same Delta table conflicts on the _delta_log entries, so serialise writers per table. lakeFS also ships a Delta export hook, with change-data-feed support since 1.77, that publishes a table's state to a location external readers can use.

Deploying for real

The quickstart flags (--quickstart, --local-settings) use an embedded database, a local blockstore and fixed credentials. They are for learning. The server refuses to combine quickstart with real settings and exits with FATAL: quickstart mode can only run with local settings.

A production configuration needs four decisions: where metadata lives, where data lives, how secrets arrive, and how it scales.

logging:
  format: json
  level: INFO
database:
  type: postgres
  postgres:
    connection_string: "postgres://lakefs:PASSWORD@pg.example.com:5432/lakefs?sslmode=require"
    max_open_connections: 25
auth:
  encrypt:
    secret_key: "<long random string, kept forever>"
blockstore:
  type: s3
  s3:
    region: me-south-1
  signing:
    secret_key: "<another long random string>"

Notes on each piece. PostgreSQL 11 or later is the on-premise baseline, and the sizing guide suggests sizing memory around the active dataset and starting KV storage at about 10 GiB. auth.encrypt.secret_key encrypts stored credentials: if you lose or change it, existing credentials become unusable, so store it in a secret manager and never regenerate it casually. blockstore.signing.secret_key should also be set in production. Choosing a bucket region near your users matters for Gulf and Egyptian workloads because of data-residency rules: the namespace is the location of your data, so create the bucket in the region your regulator expects before creating repositories.

Configuration precedence

Config comes from a file, found in this order when --config is not given: ./config.yaml, $HOME/lakefs/config.yaml, /etc/lakefs/config.yaml, then $HOME/.lakefs.yaml. Every key also maps to an environment variable: prefix LAKEFS_, upper-case, dots become underscores, so blockstore.s3.region becomes LAKEFS_BLOCKSTORE_S3_REGION. Environment variables override the file, which is how you keep secrets out of the YAML.

Kubernetes

The Helm chart is the standard route on Kubernetes and works with Helm as any other chart does:

helm repo add lakefs https://charts.lakefs.io
helm install my-lakefs lakefs/lakefs -f conf-values.yaml

existingSecret: lakefs-secrets
secretKeys:
  databaseConnectionString: database-connection-string
  authEncryptSecretKey: auth-encrypt-secret-key
lakefsConfig: |
  blockstore:
    type: s3
    s3:
      region: me-south-1

The chart's default lakefsConfig sets database.type: local and blockstore.type: local, which is for development only, and useDevPostgres: true starts a throwaway Postgres pod. Neither belongs in production. Use IRSA or an instance profile so lakeFS accesses the bucket through a role rather than static keys; the required permissions on the namespace prefix are s3:GetObject, s3:PutObject, s3:AbortMultipartUpload and s3:ListMultipartUploadParts, plus s3:ListBucket and s3:GetBucketLocation on the bucket. The error failed to create repository: failed to access storage is almost always one of these missing. If you provision the bucket and role with Terraform, put the lakeFS role policy in the same module so they cannot drift.

An NGINX ingress has a default body size of 1 MiB, which makes uploads through the gateway fail with 413; raise proxy-body-size. Terminate TLS at the load balancer, and point the load balancer health check at /_health, which returns 200 when the server is up.

Scale by adding replicas. They are stateless, so the load balancer needs no stickiness; the sizing guide recommends several small instances over a few large ones.

Garbage collection and cost

lakeFS never overwrites or deletes objects as you work, so storage only grows until GC runs. Understand it before your bill teaches you.

Deleting an object in lakeFS creates a commit in which the path is absent; the bytes stay, because older commits still reference them. GC removes two categories: committed objects that are no longer reachable within a retention window, and uncommitted objects that were abandoned (a job that uploaded and never committed). Commits themselves are never removed, only their data. An object that any branch head still contains is never collected.

You define retention per branch:

{
  "default_retention_days": 14,
  "branches": [
    { "branch_id": "main", "retention_days": 90 }
  ]
}

lakectl gc set-config lakefs://my-repo -f rules.json
lakectl gc get-config lakefs://my-repo

The rule reads: an object that was deleted or replaced on a branch is retained for that branch's number of days after the change. On Community, GC runs as a Spark job you schedule:

spark-submit --class io.treeverse.gc.GarbageCollection \
  --packages org.apache.hadoop:hadoop-aws:2.7.7 \
  -c spark.hadoop.lakefs.api.url=https://lakefs.example.com:8000/api/v1 \
  -c spark.hadoop.lakefs.api.access_key=<key> \
  -c spark.hadoop.lakefs.api.secret_key=<secret> \
  -c spark.hadoop.fs.s3a.access.key=<s3-key> \
  -c spark.hadoop.fs.s3a.secret.key=<s3-secret> \
  https://treeverse-clients-us-east.s3.amazonaws.com/lakefs-spark-client/0.21.0/lakefs-spark-client_2.12-assembly-0.21.0.jar \
  my-repo us-east-1

The client jar is version 0.21.0; there are _2.12 builds for Spark 3 and _2.13 builds for Spark 4, which needs Java 17 or later. GC has two phases, mark (decide what is collectable and write a report) and sweep (delete). You can run them separately by setting spark.hadoop.lakefs.gc.do_sweep=false to mark only, then reviewing the report under <namespace>/_lakefs/retention/gc/unified/<MARK_ID>/, and sweeping later with lakefs.gc.do_mark=false and lakefs.gc.mark_id=<ID>. The two-step form is the safe way to run GC for the first time. It is not supported on the local blockstore. On Azure, blob soft-delete must be disabled for GC to actually remove data; on GCP the service account needs the Storage Object User role.

After GC, a read of a collected object returns 410 Gone. That is expected, and it tells you a reader held an old commit past the retention window. Fix it by lengthening retention for branches whose history you must read, or by re-materialising the data.

What drives cost

Branching is free. The real drivers are churn and stale branches. A nightly job that rewrites a 500 GB table adds 500 GB a day to storage until retention expires it. A forgotten experiment branch that holds old versions keeps those objects alive indefinitely. Levers, in order of effect: shorter retention on non-main branches, deleting merged and abandoned branches, writing only changed partitions rather than whole tables, and importing instead of copying.

Security and the 1.87 changes

Version 1.87 changed what lakeFS Community is. If you operate or evaluate it, three things matter: the model of identity, the licence, and the upgrade path.

One administrator. PR #10532 removed the pluggable IAM integration and the ACL reference server. Community now runs with a single administrator, holding a single key pair. Everyone and everything uses those credentials: lakectl, SDKs, the UI login, and SigV4 to the gateway. There are no additional users, groups, policies or SSO in Community; those belong to lakeFS Enterprise and Cloud editions, which have their own version numbers, so do not mix the two. The UI pages for users, groups and credentials are gone, and lakectl auth groups and similar commands against a Community server return not implemented. Trying to create a second credential gives exceeded number of allowed credentials: invalid request.

The practical consequences follow. There is no per-user audit trail identity and no per-repository permission in Community, so treat the credential like a root key. Keep it in Vault or your cloud secret manager, inject it per workload, and rotate by re-running lakefs superuser. If you need multiple teams with different access to different repositories, that is a requirement for Enterprise RBAC, not something to approximate with clever repository names.

The licence. From 1.87 the source is under the Business Source License 1.1, previously Apache 2.0. The Additional Use Grant permits production use of the unmodified software for your organisation's internal business purposes. You may not host it for, or offer it to, third parties. Configuration through documented means does not count as modification. Each version converts to Apache 2.0 four years after its first BSL publication. If you were planning to run lakeFS as a managed service for customers, this is a blocker; take it to whoever handles licensing. One documentation FAQ still says Apache 2.0, and is stale.

Upgrading to 1.87. The release notes carry an explicit warning: if your deployment relies on a custom IAM plugin or the ACL reference server, migrate to a supported alternative first. Otherwise expect a startup guard. A server with data but no administrator of its own refuses to start, with a message beginning lakeFS has no administrator of its own. The fix is to stop the server and run:

lakefs superuser --user-name admin \
  --access-key-id <existing-key> --secret-access-key <existing-secret>

Supplying the existing key pair keeps every client working; omitting both generates new keys and every client must be updated. A related message, cannot migrate existing user to basic auth mode!, appears when several users exist from an earlier setup and the server cannot choose; lakefs superuser with the chosen user's keys resolves it. Also remove the deprecated auth.api.*, auth.oidc.*, auth.remote_authenticator.* and related keys; they are accepted but ignored with a warning per key. The keys still in force are auth.encrypt.secret_key, auth.cache.*, auth.login_duration, auth.logout_redirect_url and auth.ui_config.login_failed_message. On the client side, lakectl stopped supporting the AWS IAM credentials provider; old .lakectl.yaml blocks print a deprecation warning.

Stay current. Recent releases fixed real vulnerabilities: S3 gateway replay protection (1.75), a path traversal in the local block adapter (1.77), a dependency CVE (1.82), and repository listing without permission through the gateway (1.86). Two versions are to be avoided: 1.48.0 (squash-merge on by default) and 1.74.1 (lakectl stopped defaulting the server endpoint, fixed in behaviour by 1.74.2).

Monitoring, backup and debugging

Metrics and logs

The server exposes Prometheus metrics at /metrics on the same port. Scrape them with Prometheus and chart them in Grafana. The series worth an alert or a panel:

• api_request_duration_seconds and gateway_request_duration_seconds: latency for the API and the S3 gateway.
• api_requests_total: rate and error ratio by status.
• lakefs_concurrent_requests and blockstore_concurrent_operations: saturation.
• kv_request_duration_seconds and pgxpool_*: the database, which is the usual culprit when everything slows together.
• s3_operation_duration_seconds (and the gs_ and azure_ equivalents): the object store.

sum by (operation) (histogram_quantile(0.99, rate(api_request_duration_seconds_bucket[1m])))

Use logging.format: json in production. The audit log level is separate (audit_log_level, default DEBUG); if you set it lower in severity than the main level you silently lose audit lines. Since 1.82, requests lakeFS makes to S3 carry the User-Agent lakefs/<version>, so you can attribute traffic in bucket access logs.

Reading a slow system

Work from the outside in. A slow merge points at the KV store, since merge touches ranges and staged entries, so check pgxpool_* saturation and max_open_connections. Slow reads through the gateway point at lakeFS bandwidth or object-store latency. Slow listing on huge prefixes is expected: listing is a range scan, and the docs benchmark listing a thousand entries at about 75 ms. If everything is slow after a burst of uncommitted writes, suspect KV growth.

Backup and restore

Your data is in the bucket and your metadata is split between the bucket and the KV store, so a backup covers both. lakeFS has a plumbing command for the repository state:

lakectl refs-dump lakefs://my-repo > refs_manifest.json
# restore into a bare repository
lakectl repo create-bare lakefs://restored-repo s3://backup-bucket/restored-repo
lakectl refs-restore lakefs://restored-repo --manifest refs_manifest.json

A manifest captures commits, branches and tags; you still copy the data objects. Also snapshot the KV database regularly, and test a restore. An untested backup is a hope. Also note that lakectl refs-dump and refs-restore are listed as undocumented plumbing in the CLI reference, so pin the lakectl version you rehearse with.

Two diagnostics people miss

lakectl doctor checks your client configuration and connectivity, and is the first thing to run when a workstation fails. lakefs flare builds a diagnostic bundle for support cases, redacting secrets. For a specific failing call, re-run lakectl with --log-level debug to see the request.

Testing and CI

Data versioning makes testing pipelines pleasant, because a test can run against a real copy of production data at no storage cost.

Branch per pull request. In CI (GitHub Actions or your runner of choice), create a branch from main named for the change, run the pipeline with its output path set to that branch, run assertions, and delete the branch at the end whatever the result. A production-sized dataset is available instantly because the branch is zero-copy.

BRANCH="ci-${GITHUB_RUN_ID}"
lakectl branch create "lakefs://my-repo/${BRANCH}" -s lakefs://my-repo/main
trap 'lakectl branch delete "lakefs://my-repo/${BRANCH}" -y' EXIT
python run_pipeline.py --output "lakefs://my-repo/${BRANCH}/out/"
python assert_quality.py --path "lakefs://my-repo/${BRANCH}/out/"

The trap matters: without it, failed runs leak branches, and leaked branches keep objects alive and inflate storage until you clean up. The CI credentials are the single administrator key in Community, so store them as CI secrets and avoid printing them.

Pin test inputs. Reference a tag for golden datasets so tests do not drift when main moves.

Validate your actions. Run lakectl actions validate on every file under _lakefs_actions/ in CI, since hooks are otherwise only checked when they fire.

Load testing. lakectl ships an abuse command family (for example abuse random-read, abuse commit, abuse merge) that the sizing guide's benchmarks used. It is listed as undocumented plumbing, so check --help for your version before relying on flags, and point it at a test installation, never production.

Putting it all together

A complete pattern for a daily feature pipeline ties the sections together.

1. Provision. PostgreSQL and a bucket in your regulator's region, created with Terraform. lakeFS deployed by Helm with existingSecret, IRSA for bucket access, /_health on the load balancer.
2. Protect. lakectl branch-protect add lakefs://features 'main', plus an _lakefs_actions/ pre-merge hook that calls your validation service.
3. Run. The orchestrator creates run-<date> from main, the Spark job writes to lakefs://features/run-<date>/daily/ through lakeFSFS, and the validation step reads the same branch.
4. Merge. On success, merge into main; the pre-merge hook is the gate. On failure, leave the branch for diagnosis and alert.
5. Pin. Record the merge commit ID with the model run in your experiment tracker, and tag releases used for training.
6. Clean. A weekly job deletes merged and expired run branches; a scheduled GC, run mark-only first, reclaims the space.
7. Observe. Prometheus alerts on p99 API latency, the error ratio, and KV pool saturation.
8. Recover. A nightly refs-dump and KV snapshot, with a restore rehearsed quarterly.

lakectl branch create lakefs://features/run-2026-10-01 -s lakefs://features/main
# ... job writes, validations pass ...
lakectl merge lakefs://features/run-2026-10-01 lakefs://features/main -m "Daily features 2026-10-01"
lakectl tag create lakefs://features/train-2026-10-01 lakefs://features/main
lakectl branch delete lakefs://features/run-2026-10-01 -y

Where you are now

Before moving on, it helps to name the habits that carry over from every section. Pin what you read. Commit small. Gate what reaches main. Schedule the cleanup that nothing else will do for you. Write down how your installation is configured, so the next engineer does not have to guess.

You can now predict lakeFS behaviour from its architecture: branches are pointers, commits cost what you changed, bytes flow through the gateway or directly via pre-signed URLs, and storage only shrinks when GC runs. You can gate merges with protection and hooks, write atomic updates in Python, deploy on Kubernetes with the right secrets, reason about retention and cost, and handle the 1.87 licence and identity changes without surprise.

The next level treats lakeFS as a platform for other teams: failure modes at scale, which component breaks first, the multi-team question that Community cannot answer, and when a different tool such as DVC fits better.

Sources

• lakeFS documentation
• Architecture
• Internals
• Performance best practices
• Sizing guide
• Editions
• Importing data
• Protecting branches
• Hooks
• Webhooks
• Lua hooks
• Local checkouts
• Garbage collection
• Upgrading
• Monitoring
• Backup and restore
• Troubleshooting
• Configuration reference
• lakectl CLI reference
• S3 gateway reference
• Spark client reference
• Python SDK: transactions
• Python SDK: lakefs-spec
• Security overview
• Pre-signed URLs
• lakeFS v1.87.0 release notes
• lakeFS licence (BSL 1.1)
• Helm chart

How mid-level lakeFS interviews work

At this level nobody asks you to define a branch. Interviewers give you a situation, such as a merge that fails, a bill that grew, or an upgrade that will not start, and watch how you reason. They want to hear you name the mechanism, pick the first thing to check, and say what you would not do. This page is organised the same way: scenarios first, then trade-offs, then what separates a strong answer from a weak one. Versions quoted are lakeFS 1.87.

Scenario questions

1. After upgrading to 1.87 the server will not start

What a strong answer says. Read the log line first. Since 1.87 a server whose installation holds data but has no administrator of its own refuses to start, and the message begins lakeFS has no administrator of its own. The cause is that 1.87 removed the pluggable IAM integration and the ACL reference server, so Community now runs with a single administrator and single key pair. Installations whose users lived in an external service have none in the built-in store.

The fix is to stop the server and run lakefs superuser --user-name admin, adding --access-key-id and --secret-access-key with the existing values if clients must keep working. Without those flags you get fresh keys and every pipeline breaks until updated. Then remove the deprecated auth.api.*, auth.oidc.* and auth.remote_authenticator.* keys, which are ignored with a warning each.

What a weak answer misses. It says "roll back" without explaining why, or it tries to recreate users through the UI, which no longer has those pages. A strong candidate adds the lesson: the release notes warned to migrate off custom IAM before upgrading, so the real failure was not reading them, and the upgrade process should include a rehearsal on a copy.

2. Storage cost has doubled and nobody added data

Start by asking what the data churn looks like, because lakeFS never deletes bytes as you work. A job that rewrites a large table every night adds a full copy per run, and old copies stay until garbage collection removes them. So the checks are:

1. Is GC configured at all? lakectl gc get-config lakefs://repo shows the retention rules, and on Community GC is a Spark job someone must schedule. A missing schedule is the most common answer.
2. Are there stale branches? An object that is in any branch head is never collected, so a forgotten experiment branch pins old versions.
3. Are retention rules too generous, for example default_retention_days far above what anyone reads back?
4. Are abandoned uncommitted writes piling up? GC also handles those, but only if it runs.

For the fix, run GC mark-only first (spark.hadoop.lakefs.gc.do_sweep=false), inspect the report under _lakefs/retention/gc/unified/<MARK_ID>/, then sweep. Follow up with a policy that deletes merged run branches automatically and writes only changed partitions.

A good candidate also flags what not to do: do not delete objects directly in the bucket. lakeFS would still reference them and reads would fail.

3. A reader gets HTTP 410 Gone

410 means the object was physically removed, by GC or externally, while a commit still references it. The reader is almost certainly using an old commit or a branch that has not moved past the retention window. Ask which ref the reader uses. If it pinned an old commit, either the retention is shorter than how far back people read, or the job should not pin commits older than the window. If it reads a branch head, then something is wrong: a head object should never be collected, so look for an external deletion or an imported object whose source was removed.

That last case deserves mention because it is subtle. Imported objects sit outside the storage namespace and are never touched by GC, but they do depend on the source bucket. If a lifecycle rule on the source expires them, lakeFS still lists them and reads fail at the object store.

4. Merges are getting slow

The mechanism answers it. Merge cost scales with the size of the change, not the repository, so a slow merge means either a large change or a slow dependency. Check in order:

• How big was the change? A merge that touches millions of objects is legitimately slow. The fix is smaller, more frequent commits and merges.
• The KV store. Check kv_request_duration_seconds and pgxpool_* saturation, and whether max_open_connections (default 25) is exhausted. Uncommitted writes also live there, at roughly 150 MiB per 100,000 entries.
• Hooks. A synchronous pre-merge Lua or webhook hook adds its own latency; the webhook default timeout is a minute.
• Cache. A small committed.local_cache on a busy replica forces refetching SSTables from the bucket.

A weak answer says "add replicas". Replicas help request throughput but do not speed a single merge if the database is the bottleneck.

5. A merge fails with conflict found

Explain what a conflict is in lakeFS: the same object changed differently on both sides since the merge base, or changed on one side and deleted on the other. The comparison is per whole object, so lakeFS cannot merge inside a Parquet file. Options are to fix on the feature branch and merge again, or use --strategy source-wins or dest-wins, which resolves every conflict in that merge the same way. A strong candidate then asks why two writers touched the same object, since the real fix is often to partition outputs so concurrent jobs write different paths, or to serialise jobs per table.

6. A bad commit reached main and a dashboard is wrong

Revert, do not reset. lakectl branch revert lakefs://repo/main <commit-id> adds a commit that undoes the change and preserves history; for a merge commit add -m 1 to keep the destination side. The dashboard recovers as soon as the revert lands, because readers follow main. Then address cause: a pre-merge hook should have caught it, so add the validation that would have.

A strong answer notes that revert does not erase the bytes. If the bad data was sensitive, you must also remove references and run GC. It also notes that the branch must be clean: uncommitted changes (dirty branch) means commit or reset first.

7. Uploads through the gateway fail behind our ingress

If small files work and larger ones return 413, the ingress body limit is the cause; NGINX defaults to 1 MiB, so raise proxy-body-size. If UI previews fail while lakectl works, the UI is probably using pre-signed URLs without CORS on the bucket, and the bucket needs to allow GET, PUT and HEAD from the lakeFS origin and expose ETag. If SignatureDoesNotMatch appears, check clock skew, since expiry is validated, and the signing region, which defaults to us-east-1.

8. lakectl returns unsupported protocol scheme ""

The endpoint is unset. Since 1.74.1 lakectl no longer defaults server.endpoint_url, although the generated reference still shows a default. Fix with lakectl config or LAKECTL_SERVER_ENDPOINT_URL. The interviewer is checking whether you know docs and code can disagree, and that you trust behaviour over a stale page.

9. Repository creation fails

Two distinct errors. storage namespace already in use means another repository, or leftover _lakefs data from a deleted one, occupies that prefix; lakeFS does not delete data when a repository is deleted, so pick a new prefix. failed to access storage means the server's credentials or role cannot write and read the namespace. Check the IAM policy (GetObject, PutObject, AbortMultipartUpload, ListMultipartUploadParts on the prefix, plus ListBucket and GetBucketLocation on the bucket) and the region settings.

10. The server fails with Mismatched adapter detected

Someone changed blockstore.type after repositories existed, for example from local to s3. The fix is to restore the original type. There is an --allow-mismatched-repos flag, but behaviour is undefined, so the honest answer is to use it only to clean up, never to serve traffic.

Trade-off questions

S3 gateway or native clients?

The gateway gives compatibility: any S3 tool works unmodified, at the cost of pushing all data through lakeFS, where bandwidth becomes the limit. Native clients fetch metadata from lakeFS and move bytes directly to the bucket with pre-signed URLs, so one small server supports large throughput. A strong answer chooses per workload and names the gateway's gaps: no server-side encryption parameters, SelectObject, storage classes or tagging, and listing supports only the / delimiter.

Import or copy existing data?

Import registers objects in place with no copy, which is cheap and fast, and GC never deletes imported data. The price is that lakeFS now depends on the source bucket staying intact, it is a one-time snapshot, and importing again to the same prefix overwrites. Copying gives a self-contained repository that GC manages fully, at storage and time cost. Import to migrate fast and establish history; make new writes go through lakeFS so the repository becomes self-contained over time.

Branch per run, or write straight to main?

Branch per run costs almost nothing and gives atomic publication: consumers never see partial output, failures leave an inspectable branch, and validation can run before merge. Writing to main is simpler but exposes partial state and makes recovery a revert. A strong answer says the cost of branching is metadata only, so the usual objection does not apply, and then names the actual cost: branch hygiene, since leftovers pin storage.

Pre-merge hooks or validation inside the pipeline?

Validation inside the pipeline is faster to write and easier to debug, but it trusts every pipeline to run it. A pre-merge hook on a protected branch enforces the check for everyone, including the person merging by hand. Use both: the pipeline for fast feedback, the hook as the gate. Mention hook limits: Lua and webhook hooks are synchronous, so they add to merge latency, and an Airflow hook only confirms the DAG trigger was accepted.

PostgreSQL, DynamoDB or local for the KV store?

Local is single-node and for development. PostgreSQL is the portable default and the best choice if you already run it well. DynamoDB removes database operations on AWS and scales on demand, but costs can spike with bursts, and you can switch to provisioned capacity to cap them. CosmosDB fits Azure. The decision is mostly about which database your team can already operate, since all share the same contract.

Community or Enterprise?

After 1.87 this is sharper. Community has one administrator and one credential, so there is no per-user identity, no per-repository permission and no SSO. If a single team trusts a shared credential kept in a secret manager, Community is enough. If several teams need separation, audit identity, SSO or multi-tenancy, that is Enterprise or Cloud territory, and a strong answer says so early instead of inventing workarounds. Include the licence: the Business Source License permits internal business use of the unmodified software, and not offering it to third parties.

lakeFS or DVC?

DVC versions data alongside Git with pointer files and works best for a small team and modest datasets. lakeFS is a service over the bucket with atomic branches, server-side merges and hooks, and suits shared data lakes and pipelines. See DVC for the other side.

Questions with short model answers

Why is creating a branch instant? It writes a single pointer in the KV store. No data or metadata files are copied; commits share immutable ranges.

What is a storage namespace, and can two repositories share one? It is the object-store location assigned to a repository; all its data and metadata go there. Two repositories cannot share one.

What does a pre-hook failure do versus a post-hook failure? A failed pre-hook aborts the operation. A failed post-hook does not roll back: the merge or commit already happened.

Where do hook definitions live, and why does that matter? In _lakefs_actions/ inside the repository, so they are versioned and reviewed like data, and the merge uses the source branch's rules. It also means you must protect who can change them.

How do you make a job reproducible? Pin a tag or commit ID, and record that ID with the output or model.

What do you back up? The bucket data, the repository refs via lakectl refs-dump, and the KV database. Restore needs a bare repository and refs-restore.

How do you run GC safely? Mark first, inspect the report, then sweep; set retention per branch; confirm the storage type is supported (not local).

What are _lakefs/ and the physical address? _lakefs/ holds metadata under the namespace, and the physical address is where an object's bytes actually live. Several logical paths can map to one address.

What is a transaction in the Python SDK? An ephemeral branch that is committed and merged on success, and discarded on exception, so a multi-file change is atomic.

What is lakectl local for? Syncing a lakeFS path with a local folder, and recording the link in .lakefs_ref.yaml so a Git commit can pair with a data commit.

What a strong answer covers that a weak one misses

• It names the mechanism. "Merge cost follows change size because commits reuse immutable ranges" beats "lakeFS is fast".
• It starts with evidence. Log line, metric, config key: a strong answer reads before it changes anything.
• It knows where state lives. Object store for data and committed metadata, KV store for branch heads and staging. Many incidents are explained by that one fact.
• It distinguishes revert from reset from erasure. Weak answers treat them as interchangeable.
• It knows what GC protects and what it does not. Imported objects, branch-head objects and commits are outside its reach.
• It knows the 1.87 facts. Single administrator, lakefs superuser, the BSL, deprecated auth.* keys, and the avoided releases (1.48.0 and 1.74.1).
• It separates Community and Enterprise honestly. It does not promise RBAC on Community.
• It mentions operational hygiene. Branch cleanup, scheduled GC, tested restores, secrets in a manager.

Phrases that signal trouble

"We just turn off the hooks when they get in the way" suggests no governance. "Delete the files from the bucket" shows the candidate does not know lakeFS would still reference them. "Use --allow-mismatched-repos" as a fix, not cleanup, is a red flag. So is "reset main", which silently discards work on a shared branch, and in fact protected branches will not permit it.

Quick questions to ask back

Interviews run both ways, and good questions show experience. Ask which KV store they run and who owns it, how often GC runs and whether it has ever been run in mark-only mode, whether branch protection and pre-merge hooks guard their main branch, what happens to run branches after a failure, whether anyone has rehearsed a restore, and how they handled the 1.87 change if they are on Community. The answers tell you whether the team treats lakeFS as infrastructure or as a side experiment.

A ninety-second self-test

1. Why is branch creation constant time?
2. What is the exact cause of 410 Gone, and what are two ways to prevent it?
3. What does -m 1 mean when reverting?
4. Which component stores uncommitted changes, and what happens if a job writes ten million objects and never commits?
5. When does a post-merge hook failure undo the merge?
6. What permissions does lakeFS need on the bucket prefix?
7. Why run GC mark-only first?
8. What must you do to a config that still contains auth.oidc in 1.87?
9. What is the difference between a storage namespace conflict and an access failure when creating a repository?
10. In one sentence, when would you refuse to recommend Community?

Answers in order: branches are a pointer write in the KV store; GC removed an object a commit still references, prevented by longer retention or by readers not pinning commits older than the window; it selects parent 1, the destination side, of a merge commit; the KV store, and it pressures the database and GC will only clean it if it runs; never; GetObject, PutObject, AbortMultipartUpload and ListMultipartUploadParts on the prefix plus ListBucket and GetBucketLocation; because sweep deletes data irreversibly and a rule typo can over-collect; remove it, since it is ignored with a warning; "already in use" means the prefix is taken, "failed to access storage" means credentials or policy; when several teams need separate permissions, SSO or audit identity.

If you can answer all ten and explain the reasoning, you are ready for the senior material on running lakeFS as a platform.

Read this first

Mid-level mistakes with lakeFS are rarely typos. They are a wrong mental model that works for months and then costs money or data: a cost that only grows, a branch nobody cleaned up, a pipeline that reads main while it changes. This page lists the errors you will actually meet at lakeFS 1.87, the practices that pay off most, cards to practise on a dev server, and recipes for day-two operations. It pairs with the mid-level detailed guide, which explains the mechanisms.

Errors at this level

Each entry gives the symptom, the cause, and the fix.

The server refuses to start after upgrading to 1.87

Symptom. The log says lakeFS has no administrator of its own, then tells you to run lakefs superuser.

Cause. Version 1.87 removed pluggable IAM and the ACL reference server. Community runs with one administrator. An installation with data but no built-in administrator is refused on purpose.

Fix. Stop every instance. Run lakefs superuser --user-name admin, adding --access-key-id and --secret-access-key with your existing values if clients must keep working. Start again. Afterwards remove deprecated auth.api.*, auth.oidc.* and auth.remote_authenticator.* keys, which are accepted but ignored with a warning for each.

cannot migrate existing user to basic auth mode!

Cause. Several users exist from an earlier multi-user setup and the server cannot choose which becomes the administrator.

Fix. Run lakefs superuser with the chosen user's name and keys, following lakefs superuser -h.

exceeded number of allowed credentials: invalid request

Cause. You tried to create a second credential pair. Community allows exactly one.

Fix. Replace the credential with lakefs superuser, or move to Enterprise if you need per-person credentials. Do not try to work around it by sharing database edits.

unsupported protocol scheme "" from lakectl

Cause. No server.endpoint_url. Since 1.74.1 lakectl has no default, although the generated reference still claims one.

Fix. Run lakectl config, or export LAKECTL_SERVER_ENDPOINT_URL. You may omit /api/v1, as lakectl appends it. Note that the Hadoop client, lakeFSFS, needs /api/v1 spelled out.

storage namespace already in use

Cause. The prefix is taken by another repository, or by leftover _lakefs data from a repository you deleted. lakeFS does not delete data when you delete a repository.

Fix. Choose a fresh prefix. If the old data is truly disposable, delete it in the bucket first, knowing nothing in lakeFS will stop you.

failed to access storage when creating a repository

Cause. The lakeFS role or keys cannot write and read the namespace, or the region is wrong.

Fix. Grant s3:GetObject, s3:PutObject, s3:AbortMultipartUpload and s3:ListMultipartUploadParts on the prefix, plus s3:ListBucket and s3:GetBucketLocation on the bucket. Check blockstore.s3.region. The server log entry Could not access storage namespace carries a reason field worth reading.

Mismatched adapter detected

Cause. blockstore.type changed after repositories existed.

Fix. Restore the original type. --allow-mismatched-repos exists, but behaviour is undefined, so use it only for cleanup.

cannot write to protected branch

Cause. A branch protection rule matches. This is the feature working.

Fix. Write to a feature branch and merge. Only merges and reverts are allowed onto a protected branch.

conflict found

Cause. The same object changed differently on both branches, or changed on one and deleted on the other. lakeFS compares whole objects.

Fix. Fix the content on the feature branch and merge again. Use --strategy source-wins or dest-wins only when you know one side is authoritative, because they apply to every conflict in the merge.

uncommitted changes (dirty branch)

Cause. Revert or cherry-pick needs a clean branch.

Fix. Commit or run lakectl branch reset, then retry.

must specify 1-based parent number for reverting merge commit

Fix. Add -m 1 to keep the destination side, which is the first parent.

HTTP 410 Gone on read

Cause. GC removed an object that an old commit still references, or an external process deleted it, or an imported object's source was removed.

Fix. Lengthen retention for branches whose history people read, or stop readers pinning commits older than the window. For imported data, protect the source bucket from lifecycle deletion.

Uploads fail with 413

Cause. An NGINX ingress limits bodies to 1 MiB by default.

Fix. Raise proxy-body-size on the ingress in front of lakeFS.

UI preview fails but lakectl works

Cause. Pre-signed URLs are enabled for the UI (disable_pre_signed_ui: false) but the bucket has no CORS rule.

Fix. Allow GET, PUT and HEAD from the lakeFS origin and expose ETag.

SignatureDoesNotMatch through the gateway

Cause. Clock skew, since expiry is validated; a wrong signing region, which defaults to us-east-1; or virtual-host addressing without gateways.s3.domain_name.

Fix. Sync clocks with NTP, set the client region to us-east-1, and use path-style addressing.

A hook never seems to run, then fails at the worst moment

Cause. Action files are only validated when their event fires, so a typo in on: is invisible until a merge.

Fix. Run lakectl actions validate on every action file in CI.

Hook secrets render empty

Cause. Only server environment variables with the configured prefix are available. The code default of actions.env.prefix is LAKEFSACTION_.

Fix. Name the variable LAKEFSACTION_DQ_TOKEN and reference it as {{ ENV.DQ_TOKEN }}. Verify against the failing hook's log.

Lua hook cannot reach the network

Cause. actions.lua.net_http_enabled is false by default.

Fix. Enable it deliberately, or use a webhook hook instead.

Practices that pay off most, in order of leverage

1. Pin what you readRead tags or commit IDs in any job that must be reproducible, and log the commit ID with the output. A job that reads main reads a moving target.
2. Branch per run, merge on successConsumers of main never see partial data, failures leave evidence, and validation runs before publication. It costs metadata only.
3. Protect main and gate it with a pre-merge hookProtection forces merges; the hook makes each merge earn its place. Validate action files in CI.
4. Schedule GC, and run mark-only firstStorage never shrinks by itself. Set per-branch retention, review the mark report, then sweep.
5. Clean up branches automaticallyA branch holding old versions pins storage. Delete merged and expired run branches on a schedule.
6. Use native clients for volumePre-signed URLs keep bytes off the lakeFS server. Keep the gateway for tools that need it.
7. Keep the secrets in a managerStore auth.encrypt.secret_key, the signing key, the database string and the administrator key outside files. Losing the encrypt key makes stored credentials unusable.
8. Test a restoreDump refs, snapshot the KV database, and rehearse a restore into a bare repository before you need it.
9. Stay current, but read the notesSecurity fixes land regularly, and major releases change behaviour, as 1.87 did. Avoid 1.48.0 and 1.74.1.

Habits by area

Habits for merging

Treat every merge into a protected branch as a release. Write a real message, add metadata with --meta on the commit when a run ID or job name helps later, and tag the commits that downstream training or reports depend on. A tag is immutable, so a report that cites lakefs://repo/report-2026-q3/ is reproducible as long as retention keeps the data.

Habits for hooks

Keep hooks fast, because Lua and webhook hooks are synchronous and add to the latency of every commit or merge they guard. Use if: success() or failure() to shape notification hooks. Never put secrets in the action file; reference {{ ENV.NAME }} instead. Remember that the rules governing a merge come from the source branch, so review changes to _lakefs_actions/ as carefully as code.

Habits for the Python SDK

Use transact for any change that touches several files. Catch specific exceptions from lakefs.exceptions, retry only transient failures, and treat ForbiddenException on a protected branch as a design signal, not a retry case. Prefer lakefs-spec for pandas and Dask, since it uses pre-signed URLs.

Practice cards

1. Prove zero-copyUpload a file, commit, branch, and run lakectl fs stat on both branches. The physical address is identical. Then list the namespace and find _lakefs/.
2. Cause a conflict on purposeChange one object differently on two branches, merge both, and read conflict found. Retry with --strategy dest-wins and check which content wins.
3. Revert a mergeMerge a branch into main, then run lakectl branch revert with -m 1. Read lakectl log and see that history is added to, not rewritten.
4. Build a gateProtect main, add a pre-merge webhook that returns 500, and watch the merge fail. Use actions runs describe to find the hook. Flip it to 200 and merge.
5. Atomic updateWrite a Python transaction that uploads two files and then raises. Confirm neither file appears and no ephemeral branch remains.
6. Run GC safelySet a one-day retention, delete a branch with unique data, run GC with do_sweep=false, inspect the mark report, then sweep.
7. Survive an upgradeOn a scratch database, simulate a missing administrator and recover with lakefs superuser, keeping existing keys so a client still works.
8. Restore from nothingDump refs, create a bare repository, restore, and compare lakectl log between the original and the copy.

Operational recipes

Health, identity and connectivity

curl -i http://localhost:8000/_health     # 200 means up; use for load balancer checks
lakectl identity                          # who am I talking to lakeFS as
lakectl doctor                            # check config, credentials, connectivity
lakefs --version ; lakectl --version      # both should match the server release

A clean CI branch

BRANCH="ci-${RUN_ID}"
lakectl branch create "lakefs://my-repo/${BRANCH}" -s lakefs://my-repo/main
trap 'lakectl branch delete "lakefs://my-repo/${BRANCH}" -y' EXIT
# run the pipeline against lakefs://my-repo/${BRANCH}/...

Publish a release

lakectl merge lakefs://my-repo/run-2026-10-01 lakefs://my-repo/main -m "Daily 2026-10-01"
lakectl tag create lakefs://my-repo/release-2026-10-01 lakefs://my-repo/main
lakectl branch delete lakefs://my-repo/run-2026-10-01 -y

Find out what changed

lakectl diff lakefs://my-repo/main lakefs://my-repo/feature --prefix features/
lakectl log lakefs://my-repo/main --amount 20 --first-parent
lakectl show commit lakefs://my-repo/<commit-id>

The --first-parent flag keeps the log readable when many branches have merged. Use --prefixes and --since to narrow it to the data you care about.

Check protection and recent hook runs

lakectl branch-protect list lakefs://my-repo
lakectl actions runs list lakefs://my-repo --branch main
lakectl actions runs describe lakefs://my-repo <run-id>

Inspect and set GC rules

lakectl gc get-config lakefs://my-repo
lakectl gc set-config lakefs://my-repo -f rules.json

Back up repository state

lakectl refs-dump lakefs://my-repo > refs_manifest.json
lakectl repo create-bare lakefs://restored s3://backup-bucket/restored
lakectl refs-restore lakefs://restored --manifest refs_manifest.json

The manifest holds commits, branches and tags. The data objects and the KV database are separate backups, and the restore target must be bare. These commands are listed as plumbing in the CLI reference, so pin the lakectl version you rehearse with.

Environment variables worth knowing

Metrics to alert on

Alert on the p99 of api_request_duration_seconds, the error ratio from api_requests_total, saturation in lakefs_concurrent_requests and pgxpool_*, and latency in s3_operation_duration_seconds. A query to start from:

sum by (operation) (histogram_quantile(0.99, rate(api_request_duration_seconds_bucket[1m])))

A checklist before you call a lakeFS setup production-ready

• PostgreSQL (or DynamoDB or CosmosDB) is the KV store, not local, and is backed up.
• auth.encrypt.secret_key and blockstore.signing.secret_key are set, stored in a secret manager, and never regenerated.
• The bucket access uses a role (IRSA or an instance profile) with the listed permissions, not static keys where avoidable.
• The bucket is in the region your regulator expects for Gulf or Egyptian data.
• The load balancer health check is /_health, TLS terminates at the load balancer, and the ingress body limit is raised.
• main is protected and a pre-merge hook validates it; action files are validated in CI.
• GC is scheduled, retention is set per branch, and the first run was mark-only.
• A job removes merged and expired branches.
• Metrics are scraped and alerts exist; logs are JSON.
• A refs dump and KV snapshot run on a schedule, and a restore has been rehearsed.
• The team has read the 1.87 notes: one administrator, the Business Source License, and the deprecated auth.* keys.

Work through the practice cards on a dev server and this list on your real one. Each item removes a class of incident, and together they are most of what separates using lakeFS from using it well. The Docker guide's mid-level tips follow the same pattern for the container side of this stack.

Reading a failure quickly

When something breaks, resist changing settings until you know which layer failed. A short routine saves hours. First, decide whether the failure is on the client or the server: run lakectl doctor, and if it passes, the problem is in the operation, not your setup. Second, read the exact message, because lakeFS errors are specific. not found names what is missing, conflict names a merge, protected branch names a rule, and not implemented on an auth command means you are talking to Community. Third, check the server log in JSON format for the same request. Fourth, only then look at dependencies: the KV database through pgxpool_*, the bucket through s3_operation_duration_seconds, and any hook through actions runs describe.

Keep a scratch repository on a throwaway namespace where you can reproduce a problem safely. Most of the errors above take two minutes to reproduce there, and reproducing them is the fastest way to learn what the message really means. If you need to open a support case, lakefs flare produces a diagnostic bundle with secrets redacted, and you should attach the lakeFS and lakectl versions, since behaviour changed in several recent releases.

Finally, keep notes on your own installation: which KV store, which blockstore region, which retention rules, and which version you upgraded from. Mid-level engineers are the people a team turns to when the platform misbehaves, and a one-page record of how your installation is configured turns a long incident into a short one. Put that page next to the runbooks for the rest of your stack, such as the ones for Kubernetes and Prometheus, so whoever is on call finds it where they already look.

Review that page every quarter, and whenever you upgrade. Configuration drifts quietly, and the moment you most need an accurate record is the moment something has already gone wrong, so keeping it current is cheap insurance for the whole team.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further and adding the work you own when lakeFS is a platform that other teams depend on rather than a tool you use yourself.

Up to this point the questions were about making branches, commits and merges work on data. From here they are different: which component fails first when sixty people use it, what a compromised credential can reach, what happens to your installation when a release changes the licence and removes the authentication plug-ins it depended on, and how you would rebuild the whole thing from a blank account after a bad day. The facts in this guide are checked against lakeFS v1.87.0, released on 22 September 2026. That release changed more about how you run the product than any release in years, so a large part of this guide is about it.

Where this picks up

I am starting with the architecture, because every operational decision in this guide follows from knowing exactly where lakeFS keeps its state.

The architecture you are actually running

You know from earlier levels that lakeFS puts a Git-like layer over an object store. As an owner you need a more precise picture. lakeFS is one stateless Go binary listening on one port (8000 by default). That single process serves three things: the web UI, the OpenAPI server under /api/v1, and the S3-compatible gateway. Inside the process there are four components worth naming: an authentication service, Graveler (the versioning engine), a storage adapter for your object store, and the hooks engine that runs actions.

Everything that matters is stored in exactly two places, and the distinction drives your whole operating model.

The committed side is immutable. Each commit is represented by a two-level Merkle tree of SSTables, which are RocksDB-compatible sorted files. The leaves are called ranges and the root for a commit is a metarange, a range of ranges. Because these files are immutable and content-addressed, two consequences follow that you will rely on repeatedly. First, a commit or merge costs time proportional to the size of the change, not the size of the repository, because unchanged ranges are simply referenced again. Second, committed metadata is safe to cache aggressively and safe to copy: a backup of the namespace is a backup of every commit.

The mutable side is the key-value (KV) store. It holds the pointer that says what commit main currently names, every uncommitted write waiting to be committed, the credentials, and the records of action runs. The supported backends are PostgreSQL, DynamoDB and CosmosDB for production, and an embedded local store for single-node development. Redis and MemoryDB exist in Enterprise. This split is the most important fact in the architecture, so I will state its consequence plainly: lose the object store and you lose data; lose the KV store and you lose the present tense, meaning where every branch points and every uncommitted change. Your recovery plan has to treat the two differently.

The local disk of each server is only a cache. The default committed-metadata cache, committed.local_cache.size_bytes, is 1 GiB and lives under ~/lakefs/data/cache. Losing a server loses nothing, which is why servers are cattle and the database is the pet. It is also why the sizing guide recommends ephemeral NVMe for the cache rather than network-attached disks: the cache exists to avoid round trips to the object store.

The data you manage never passes through a second copy. The storage namespace you give a repository, for example s3://bucket/prefix, receives every object that lakeFS writes, committed or not, and lakeFS never modifies an object once written; it only deletes through garbage collection. Several logical paths across versions and branches can map to one physical object, and that sharing is the whole basis of zero-copy branching.

Failure modes: what breaks, and what it looks like

A platform owner should be able to name the failure before the page arrives. Because the architecture has few moving parts, the failure modes are few, and each has a recognisable signature.

The KV database is slow or down. Every branch operation, every uncommitted write and every authentication check touches the KV store. PostgreSQL connection pool exhaustion is the most common form: the server keeps at most database.postgres.max_open_connections connections (default 25) per instance, so ten instances can open 250 connections against a database that may be configured for 100. The signature is rising kv_request_duration_seconds and pgxpool_* metrics showing acquired connections pinned at the maximum, followed by timeouts in the API. On DynamoDB the signature is throttling, governed by the table's capacity mode; lakeFS creates a table named kvstore by default and does not manage its lifecycle, so capacity, backups and deletion protection are yours to configure.

The object store refuses the lakeFS identity. If the role or key lakeFS uses loses permission on the namespace, reads of metadata fail and commits cannot write new SSTables. The signature is HTTP errors from the object store surfaced through the API, and repository creation failing with failed to create repository: failed to access storage. The required permissions on the prefix are s3:GetObject, s3:PutObject, s3:AbortMultipartUpload and s3:ListMultipartUploadParts, plus s3:ListBucket and s3:GetBucketLocation on the bucket.

A pre-merge or pre-commit hook blocks everything. Hooks run synchronously for Lua and webhook types, and a failing pre-* hook aborts the operation. A webhook endpoint that is down, slow, or returning errors therefore stops every merge to every protected branch. The signature is merges failing with hook errors and an action run marked failed in lakectl actions runs list. This is a failure you created on purpose by adding a gate, and it needs an owner and a bypass procedure, which the governance section covers.

Garbage collection removes data a reader still wants. A read of an object that GC collected returns 410 Gone, even though the commit that references it still exists. The signature is a pipeline that reads an old commit or an old tag and fails on some objects. It is not a bug; it is retention doing what the rules said.

The signing or encryption secret changes. auth.encrypt.secret_key encrypts stored credentials, and if it is lost or changed, existing credentials become unusable. blockstore.signing.secret_key should be set explicitly in production. Both are configuration, both are secrets, and both are easy to regenerate accidentally in a redeploy that templates them from a random source.

The data path and what scales first

Mid-level taught you that lakeFS offers two ways to move bytes. At this level the choice stops being a convenience and becomes the single largest lever on your capacity.

Through the S3 gateway, the data flows through lakeFS. A client speaks S3 to https://lakefs.example.com, using the bucket form s3://<repo>/<branch>/<path>, and the lakeFS server streams the bytes to and from the object store. Every gigabyte read by every job crosses your lakeFS servers and your load balancer, so network bandwidth of the lakeFS fleet is the scaling factor. The gateway also implements only a subset of S3: SigV2 and SigV4, bucket listing and head, get, put, head and delete of objects, bulk delete, copy, list objects (v1 and v2, with / as the only delimiter) and multipart upload. Listing parts and listing multipart uploads are currently supported only when the underlying store is AWS S3. There is no server-side encryption request handling, no SelectObject, no storage classes and no object tagging.

With native clients, meaning lakectl, the Python SDK, the Hadoop lakeFSFS filesystem and, in commercial editions, Everest, the client asks lakeFS only for metadata and a pre-signed URL, then reads or writes directly against the object store. The documented cost is on the order of one kilobyte per request to lakeFS, and the bytes never touch it. This is the path that scales, and the one to standardise on.

The practical rule for a platform is therefore: make the native path the default and the gateway the exception. Use the gateway for tools that can only speak S3, such as R, MATLAB and some third-party engines, and size it for bandwidth when you do. For Spark, prefer lakeFSFS with fs.lakefs.impl=io.lakefs.LakeFSFileSystem over S3A through the gateway when you can, because it keeps the data path direct. Pre-signed access has a side effect worth knowing: the bucket can stay closed to users entirely, because lakeFS signs on their behalf and the expiry is controlled by blockstore.s3.pre_signed_expiry (15 minutes by default).

What breaks first, in the order I have seen teams meet it:

1. Gateway bandwidth, when heavy readers are pointed at the S3 endpoint instead of native clients.
2. Database connections and IOPS, when many replicas share a PostgreSQL instance sized for one.
3. Listing cost, when a job lists millions of objects under one prefix repeatedly; listing 1,000 prefixes is about 75 ms p90 in the documented benchmark, and deep listings multiply.
4. Hook latency, when a synchronous webhook becomes the slowest step of every merge.
5. Uncommitted write volume, when a pipeline writes hundreds of millions of objects to a branch before committing; staged entries live in the KV store at about 150 MiB per 100,000 uncommitted writes.

Notice what is not on the list: repository size and branch count. Branch creation is metadata-only and stays under roughly 30 ms p90, and commit cost follows the size of the change. The sizing guide's benchmark used about 180 million objects per commit on two c5ad.4xlarge servers with a db.m6g.2xlarge database, so the architecture is built to make size a non-issue and traffic shape the issue.

The security model and the trust boundary

The trust model changed in 1.87 and any description of it that predates that release is wrong, so I will build it from the current behaviour.

Community, from v1.87.0, has a single administrator. One user, one set of credentials (the basic authentication service allows one credential per user), and that access key and secret are what every client uses: basic authentication to the API, SigV4 to the gateway, and the only accepted login to the web UI. There is no per-user identity, no group, no policy and no single sign-on in Community any more. The release removed pluggable IAM and the ACL reference server, and the UI pages for users, groups, policies and credentials are gone. Creating a second key pair for the administrator fails with exceeded number of allowed credentials: invalid request, and group or policy operations through lakectl auth return not implemented against a Community server.

What that means for a platform owner is uncomfortable but simple. Every person and every job that uses a Community installation holds the same identity. Audit logs cannot distinguish them, revoking one person's access means rotating the shared key for everyone, and there is no way to give a team read-only access or restrict it to one repository. If your organisation needs any of those, the supported answer is Enterprise or Cloud, which provide RBAC policies (with fs:*, auth:*, ci:* and retention:* actions, and StringLike conditions since 1.80), single sign-on through OIDC and SAML, SCIM, a remote authenticator, AWS IAM role login, tenants and audit logs. The other supported answer is to stay on 1.86 or earlier for as long as you need the plug-in, accepting that you then lag on security fixes.

The second half of the trust model is the storage identity. lakeFS needs permission on the buckets that hold repositories, and the safest form of that permission is a role obtained through an instance profile, IRSA or workload identity rather than static keys. Users of native clients need no bucket permissions, because lakeFS signs the URLs for them. This is a real security benefit, but it moves the trust: whoever holds the lakeFS administrator key can read and write every byte in every namespace. Treat that key as equivalent to bucket administrator on those prefixes.

Third, the transport and edge. Terminate TLS at the load balancer. The tls.* keys exist mainly for development. Put the UI and API behind whatever network controls you already use, and remember that the gateway and the API share a port, so a network rule that blocks one blocks the other. If you use NGINX ingress on Kubernetes, raise proxy-body-size above its 1 MiB default or uploads over that size fail with 413. If the UI uses pre-signed URLs, which is controlled by disable_pre_signed_ui (true by default on S3), the bucket needs a CORS rule allowing GET, PUT and HEAD from the lakeFS origin and exposing ETag.

Fourth, keep current. The recent security record is reason enough: the gateway replay weakness fixed in 1.75, a path traversal in the local block adapter fixed in 1.77, a dependency vulnerability in the PostgreSQL driver fixed in 1.82, and a gateway repository-listing exposure fixed in 1.86 for users without fs:ListRepositories. A platform that is six months behind is carrying known advisories. Subscribe to the security advisories on the repository and treat a new advisory like any other production dependency alert.

Secrets and the supply chain

Secrets in a lakeFS deployment are few, and each has a blast radius you can write down.

The encryption secret, auth.encrypt.secret_key, is required and has a built-in development default, THIS_MUST_BE_CHANGED_IN_PRODUCTION, that only local and quickstart settings use. The signing secret, blockstore.signing.secret_key, should be set explicitly. The database connection string contains a password. Static storage keys, if you use them, and the SMTP password complete the list. The administrator's access key and secret are the most sensitive of all, because they are the identity.

Hook secrets need their own rule. Webhooks and Lua scripts often need tokens, and lakeFS lets a webhook's query_params and headers reference server environment variables with the {{ ENV.VAR }} syntax, so the secret never lives in the repository. The environment variables must carry a prefix, configured by actions.env.prefix, which defaults to LAKEFSACTION_ in the code. One docs page spells it LAKEFSACTIONS_, and that is a typo in the page, so teach the configurable key and verify the prefix on your own installation. Setting actions.env.enabled: false blocks this access entirely, actions.lua.net_http_enabled is false by default so Lua hooks cannot make outbound HTTP calls until you enable it, and actions.enabled: false disables hooks altogether. Every one of those is a switch you can use to narrow what a repository's author can make the server do.

That last point is a real trust boundary. Action files live in the repository, under _lakefs_actions/, and they are read from the branch being acted on. Anyone who can write to a branch can therefore propose code that the lakeFS server runs: a Lua script, or a webhook call carrying your environment secrets in its headers. Branch protection on the branch that supplies the actions, plus review of changes to _lakefs_actions/, is what stops a contributor from turning a data repository into remote code execution on your server.

For the supply chain, the artefacts to verify are small. The release archives ship with a checksums.txt file, so verify downloads against it. Pin container images to an exact version such as treeverse/lakefs:1.87.0 or, better, to a digest, instead of latest. The Helm chart lakefs/lakefs pins its default image.community.tag to a specific version, and the chart version and the application version move independently, so record both. The Python clients have their own pins: lakefs-sdk tracks the server version, and the high-level lakefs package (0.16.0 at the time of writing) depends on lakefs-sdk>=1.50,<2 and is published under the same Business Source License. Spark clients are pinned too: lakefs-spark-client 0.21.0 for garbage collection and io.lakefs:hadoop-lakefs-assembly:0.18.0 for the Hadoop filesystem. Those numbers are what the documentation currently names, so put them in your dependency manifest and renew them on a schedule.

The licence change, and what it means for you

From v1.87.0 lakeFS Community is released under the Business Source License 1.1 instead of Apache 2.0, with Treeverse Labs Ltd as the licensor. You own a platform, so you own this question, and you should answer it in writing before the upgrade.

The terms that matter are these. The Additional Use Grant permits production use of the unmodified Licensed Work, and only for your organisation's own internal business purposes. You may not host lakeFS for, or offer it to, third parties, whether paid or unpaid. Configuring lakeFS through its documented configuration, flags, plug-ins and interfaces does not count as modifying it. The Change Date is four years after each version is first published under the licence, and the Change License is Apache 2.0, so each version becomes open source under Apache 2.0 after four years. The FAQ page that still says lakeFS is licensed under Apache 2.0 is stale, and the editions page now says Community is free under the Business Source License.

Translate that into decisions. If you run lakeFS for your own company's data teams, including many teams across a large company, you are within the grant as long as you do not modify the code. If you are a consultancy that hosts a lakeFS instance for clients, a platform vendor embedding it in a product sold to customers, or a training company giving learners accounts on a shared installation they do not work for you on, you are in territory the grant does not cover and you should ask the licensor or use a commercial edition. If you maintain a patched fork, you have left the grant. Versions before 1.87 remain under their original licence, and the change is not retroactive, but they also do not receive new features or, eventually, fixes.

Multi-tenancy: what the tool gives you, and what you must build

Teams share a platform, and the question is how much they can see of each other. lakeFS gives you one isolation unit, the repository, and each repository has its own storage namespace. A repository's branches, commits, tags and hooks are separate from every other repository's, and two repositories cannot share a storage namespace: creating the second fails with failed to create repository: storage namespace already in use. lakeFS also does not delete data when you delete a repository, so a reused prefix can collide with leftover _lakefs data. Choose namespace prefixes that encode the owning team and never reuse them.

What a repository does not give you in Community is an access boundary, because the single administrator can reach all of them. Per-user, per-group and per-repository permissions, tenants and SCIM provisioning are Enterprise features, as is support for multiple storage backends in one installation (configured through a storage_id). The editions page describes a Team tier, limited to five users, 500 GB and 500 requests per second and adding metadata search, datasets, an Iceberg REST catalog, audit, access control and mount, and an Enterprise tier that adds multi-tenancy, asynchronous commit and merge, standalone garbage collection, configurable rate limits and a self-managed or Cloud deployment. Check the editions page when you plan, because the tier boundaries are commercial and change.

So there are three honest designs for a multi-team organisation, and you should choose one deliberately.

1. One Community installation per team or business unit. The isolation is complete, because each has its own administrator key, database and bucket prefix. The cost is operational repetition, so you automate the deployment with Terraform and Helm and accept the overhead. See the Terraform and Helm guides.
2. One shared Community installation with a trust agreement. Everyone uses one identity and you rely on convention and review. This works for a small, high-trust data team and fails the first time an auditor asks who deleted a branch.
3. One Enterprise or Cloud installation with RBAC. This is the designed answer for a shared platform, and it is the one the licence terms and the single-administrator change both push you toward.

Whichever you pick, put the tenant in the name. A repository called ml-features-prod tells a reader more than features. Repository names must start with a lowercase letter or number, contain only lowercase letters, numbers and hyphens, and be 3 to 63 characters long, so design a naming scheme inside that grammar before the first repository exists.

Cost: where the money goes and how to stop it

Branching is free in the sense that matters: creating a branch copies no data. The bill arrives from three other places.

Retained versions. Every object lakeFS ever wrote stays in the storage namespace until garbage collection removes it, and an object that is reachable from any branch head is never collected. A data scientist who branches, rewrites a three-terabyte table, commits, and abandons the branch has added three terabytes that stay until the branch is deleted and GC runs. Storage cost is therefore driven by the rate of change and by retention policy, not by the count of branches.

Uncommitted and abandoned writes. Writes to a branch that never get committed sit in the KV store and the namespace. Garbage collection can also clean unreachable uncommitted objects, which is why GC is part of cost control and not only compliance.

The controls are garbage collection rules, scheduled GC, branch hygiene and import. Retention rules live in a JSON document:

{
  "default_retention_days": 14,
  "branches": [
    { "branch_id": "main", "retention_days": 21 }
  ]
}

lakectl gc set-config lakefs://my-repo -f rules.json
lakectl gc get-config lakefs://my-repo

In Community, GC is a Spark job, so it needs a Spark environment and credentials for both lakeFS and the object store. Enterprise has a standalone GC program named lakefs-sgc that needs no Spark, and Cloud has managed GC. A typical Community invocation follows; replace the repository name and region with yours.

spark-submit --class io.treeverse.gc.GarbageCollection \
  --packages org.apache.hadoop:hadoop-aws:2.7.7 \
  -c spark.hadoop.lakefs.api.url=https://lakefs.example.com:8000/api/v1 \
  -c spark.hadoop.lakefs.api.access_key="$LAKEFS_ACCESS_KEY" \
  -c spark.hadoop.lakefs.api.secret_key="$LAKEFS_SECRET_KEY" \
  -c spark.hadoop.fs.s3a.access.key="$AWS_ACCESS_KEY_ID" \
  -c spark.hadoop.fs.s3a.secret.key="$AWS_SECRET_ACCESS_KEY" \
  https://treeverse-clients-us-east.s3.amazonaws.com/lakefs-spark-client/0.21.0/lakefs-spark-client_2.12-assembly-0.21.0.jar \
  my-repo us-east-1

Two design facts about GC shape how you run it. GC works in two phases, mark and sweep. You can run mark only with spark.hadoop.lakefs.gc.do_sweep=false, review the report that lands under <namespace>/_lakefs/retention/gc/unified/<MARK_ID>/, and later run sweep only with lakefs.gc.do_mark=false and lakefs.gc.mark_id=<ID>. Use that split the first time you run GC on a repository, so a person looks at what would be deleted before it is. Second, GC never removes commits, only the objects behind them, and it never deletes imported objects, because they live outside the namespace. Constraints: GC is not supported with the local blockstore, Azure needs blob soft-delete disabled, and on Google Cloud the service account needs the Storage Object User role. The Spark client has Scala variants, _2.12 for Spark 3 and _2.13 for Spark 4, and Spark 4 needs Java 17 or later. See the Spark guide for running the job.

The organisational half of cost control is branch hygiene. A branch that nobody owns pins every object it references. Require an owner and an expiry in the branch name or commit metadata, and run a weekly report of branches older than your policy. Together with a documented retention window, that turns cost into a decision rather than a surprise.

Upgrades and migrations

Upgrading lakeFS is usually a redeploy. The discipline is in knowing when it is not, and in having a rollback boundary written down before you start.

The standard procedure: read the release notes for the versions you are crossing, take a snapshot of the KV database, commit or discard outstanding changes that matter, and deploy the new image or binary. When a release notes a schema change, the procedure becomes: stop all instances, run lakefs migrate up with the new binary, then start the new version. An older binary cannot run against a migrated database, which is your rollback boundary: once migrate up has run, going back means restoring the snapshot. Rolling upgrades of the KV layer have been supported since 0.103.0. From a version below 0.80 you must pass through 0.80.2, and from below 0.30 through 0.30.0. lakefs migrate version reports where you are.

Keep a version blocklist. Do not deploy 1.48.0, where squash-merge was accidentally on by default (fixed in 1.48.1), and do not pin 1.74.1, which introduced the lakectl endpoint change and was followed within hours by 1.74.2, which fixed a panic when a new lakectl talks to an older server. Pinning deliberately is correct; pinning to a release with a known follow-up fix is not.

Upgrading to 1.87 is a migration, not a redeploy

This is the upgrade that will catch people. The release note says it directly: if your deployment relies on a custom IAM plugin or the ACL reference server, migrate to a supported alternative before upgrading. The steps for an installation that held data and users under the old model:

1. Decide your target. Enterprise RBAC if you need multiple identities; otherwise accept a single administrator, or stay on 1.86 or earlier.
2. Inventory the clients that use keys other than the one you will keep as the administrator. Every one of them will stop working unless you migrate the key.
3. Clean the configuration. The keys auth.api.*, auth.authentication_api.*, auth.remote_authenticator.*, auth.oidc.*, auth.cookie_auth_verification.*, auth.login_max_duration and most of auth.ui_config.* are accepted but ignored, with a warning logged for each. The live keys that remain are auth.encrypt.secret_key, auth.cache.*, auth.login_duration, auth.logout_redirect_url and auth.ui_config.login_failed_message. A config that still names the old keys will run but will mislead the next person who reads it.
4. Create or repair the administrator. A new startup guard means a server whose installation has been used but has no administrator of its own refuses to start. The message reads lakeFS has no administrator of its own, and it holds repositories: run "lakefs superuser --user-name <name>" to create one with fresh credentials, adding --access-key-id and --secret-access-key to keep a key pair your clients already use.
5. Run it with the key your clients already hold, while the server is stopped:

lakefs superuser --user-name admin \
  --access-key-id "$EXISTING_ACCESS_KEY_ID" \
  --secret-access-key "$EXISTING_SECRET_ACCESS_KEY"

6. Start the new version and verify with curl -i http://localhost:8000/_health, lakectl identity and a read through the gateway.

Related startup messages carry different meanings. A variant names auth.api.endpoint as an external authorisation service; the fix is the same command, or removing the auth.api keys on a new installation. cannot migrate existing user to basic auth mode! Please run "lakefs superuser -h" and follow the instructions on how to migrate an existing user appears when several users existed and the server cannot choose among them; run lakefs superuser with the chosen user's keys.

lakectl changed in the same release. It dropped its AWS IAM credentials provider, so credentials.provider.type and credentials.provider.aws_iam.* in ~/.lakectl.yaml are parsed but ignored, and lakectl prints a warning that AWS IAM authentication is available in Enterprise, which ships its own lakectl. Older still: since 1.74.1 lakectl has no default server endpoint. The generated CLI reference still shows a default, but the code sets none, so every client configuration must say where the server is, either in ~/.lakectl.yaml or through LAKECTL_SERVER_ENDPOINT_URL. If a fleet of CI images carries an old config file, you will find out from unsupported protocol scheme "".

Migrating data into lakeFS is a different kind of migration and usually the larger one. lakectl import registers existing objects without copying them and creates a commit. The source must not be inside a storage namespace (rejected since 1.54), the import overwrites the destination prefix, and imported objects are never touched by GC. That last point cuts both ways: it protects the source data, but it also means the bytes stay in the original location and the original bucket's lifecycle rules can delete them out from under your commits. Decide who owns the source prefix after import, and turn off any lifecycle rule that could expire it.

Backup, disaster recovery and the restore drill

Remember the split in the architecture: the object store holds data and committed metadata, the KV database holds the present. A recovery plan covers both and is tested, or it is a hope.

For the object store, enable your cloud's versioning and cross-region replication on the bucket where appropriate. Because lakeFS never modifies objects it has written, replication of the namespace is a straightforward copy. For the KV database, take snapshots or point-in-time recovery: PostgreSQL continuous archiving or managed snapshots, DynamoDB point-in-time recovery and on-demand backups, CosmosDB continuous backup. Enterprise offers mirroring and replication between installations.

lakeFS also provides a logical backup of the references, which is what makes a clean rebuild possible:

lakectl refs-dump lakefs://my-repo > refs_manifest.json

# On a rebuilt installation, create a bare repository (no initial commit or
# branch) on the copied storage namespace, then restore the references.
lakectl repo create-bare lakefs://my-repo s3://restored-bucket/my-repo
lakectl refs-restore lakefs://my-repo --manifest refs_manifest.json

These two commands are listed as undocumented plumbing in the CLI reference, so pin the lakectl version you use for them and rehearse the procedure on every upgrade. The server also has lakefs kv dump with --all and --repo options (since 1.75), which writes out the contents of the key-value store for inspection or archive.

Write a recovery objective in words a stakeholder understands. For example: recovery point of fifteen minutes for branch pointers (set by database backups), zero for committed data (set by object versioning), and recovery time of two hours for a rebuilt installation (set by how well your infrastructure code is tested). Then run the drill: build a clean installation from code, restore a database snapshot into it, point it at a replica of the bucket, and check that lakectl log for a known tag shows the commits you expect and that a known object reads back with the right checksum.

Governance: hooks, protection and pull requests as platform policy

Earlier levels taught hooks and branch protection as features. A platform owner uses them as policy that is enforced where the data changes, instead of policy enforced by a wiki page.

The shape of the governance layer is small. Branch protection rules are glob patterns: a matching branch rejects direct writes, deletes, commits and resets, so only merges (and branch reverts) can change it. The pattern is managed with lakectl branch-protect add lakefs://my-repo 'main', and violations surface as cannot write to protected branch or cannot commit to protected branch. Actions are YAML files in _lakefs_actions/ with events, optional branch filters and ordered hooks of type lua, webhook or airflow. The events include pre- and post- variants of commit, merge, branch creation and deletion, revert, tag creation and deletion, and cherry-pick, plus an experimental prepare-commit. pre-* failures abort the operation; post-* failures never roll it back. Pull requests give a reviewable diff and a merge action in the UI and API.

A pre-merge gate that protects production tables looks like this:

name: pre-merge data quality gate
on:
  pre-merge:
    branches:
      - main
hooks:
  - id: quality_check
    type: webhook
    description: Reject the merge if the quality service says no
    properties:
      url: "https://dq.example.com/lakefs/pre-merge"
      timeout: 2m
      headers:
        Authorization: "Bearer {{ ENV.DQ_TOKEN }}"

Validate a file locally with lakectl actions validate ./_lakefs_actions/pre-merge-quality.yaml, but remember that action files are otherwise validated only when an event fires, so a typo is discovered by the first person to merge. Run logs are written under _lakefs/actions/log/<runID>/, and lakectl actions runs list and lakectl actions runs describe let you inspect them.

As an owner you are designing for failure. Five rules have held up.

1. A gate needs an owner and an SLO. A webhook that blocks merges is a production dependency. Give it an on-call rotation, because its outage is a data platform outage.
2. Make the gate fast. The default webhook timeout is one minute. A quality check that takes ten minutes belongs in a post-merge hook on a staging branch, or in an orchestrated pipeline, not in a synchronous pre-merge hook.
3. Design the bypass. Decide in advance who may disable a hook or an action in an emergency, how that is recorded, and how it is re-enabled. The blunt instrument is actions.enabled: false, which affects everyone.
4. Protect the policy. Branch-protect the branch that supplies _lakefs_actions/, and require review on changes to it, for the trust-boundary reason given earlier.
5. Prefer post hooks for notifications and pre hooks for decisions. A failed Slack message should never stop a merge.

Incident playbooks

Write these down before you need them. Each follows the same pattern: confirm the symptom, find the layer, act, then record.

Playbook one: the API is slow or returning 5xx. Check /_health on each instance, then kv_request_duration_seconds and the pool metrics. If the database is the bottleneck, reduce load first (pause heavy batch jobs), then raise capacity: DynamoDB capacity, or PostgreSQL size and connection limits. If one instance is unhealthy, remove it from the load balancer; the others carry on because servers are stateless.

Playbook two: a client is rate limited. An HTTP 429 means the request rate limit was reached. Rate limits are a commercial-tier feature: 500 requests per second on the Team tier and configurable on Enterprise. lakectl retries four times by default (server.retries.max_attempts: 4), so a batch job needs backoff and bounded parallelism rather than more retries. Note that the API specification changed from 423 to 429 in 1.64.1, so clients from before that may still treat 423 as a special case.

Playbook three: a merge fails with conflict found (409). The same object changed differently on both sides, or was changed on one and deleted on the other. Conflicts are per whole object; lakeFS does not merge inside files. Resolve by correcting on a branch, or merge with --strategy source-wins or dest-wins, which applies to all conflicts in that merge, so use it only when you understand every conflicting path.

Playbook four: bad data reached main. This is the incident lakeFS is designed for. Do not edit in place. Identify the commit with lakectl log, then revert it: lakectl branch revert lakefs://my-repo/main <commit>, adding -m 1 if it is a merge commit, because reverting a merge commit requires a parent number. Consumers pinned to a tag or commit ID were never affected. Write the post-incident change: a pre-merge gate that would have caught this.

Playbook five: readers get 410 Gone. Confirm that GC ran, read the retention rules with lakectl gc get-config, and find the commit the reader pinned. If the data is wanted, you will need to restore from a bucket-level version if you have one; lakeFS cannot recover objects GC swept. Then fix the retention window or the pinned reference.

Playbook six: repository creation fails. failed to create repository: failed to access storage means lakeFS's identity cannot write or read the namespace: check the role, the bucket policy, blockstore.s3.region and discover_bucket_region. storage namespace already in use means another repository or leftover _lakefs data occupies the prefix; choose a new prefix.

Playbook seven: the server will not start after an upgrade. Read the first error line. The administrator guard message means you have not run lakefs superuser. A mismatched adapter message means the blockstore type changed. If the database was migrated and you must go back, restore the snapshot taken before migrate up, because the older binary cannot read the migrated schema.

Playbook eight: a credential leaked. On Community there is one identity, so rotation affects everyone. Run lakefs superuser with new credentials while the server is stopped, update the secret manager, redeploy clients, and review access logs at the object store for the window since exposure. After the incident, raise the question of Enterprise RBAC, because this incident is what the single-identity design costs.

After each incident, write the timeline, the detection gap and one change in the platform that makes the next one cheaper.

Where lakeFS stops

An honest platform owner knows when the tool is the wrong tool. lakeFS is a versioning and branching layer for object storage, and it is not several things people sometimes assume.

It is not a table format or a query engine. It versions files. If you need ACID tables with schema evolution and time travel inside a table, use Delta Lake or Iceberg on top, and use lakeFS to branch and protect the storage underneath them; lakeFS can export Delta tables and supports change data feed export since 1.77, and the Iceberg REST catalog is a commercial feature.

It is not a per-file merge engine. Conflicts are whole-object, so two people editing different rows of the same Parquet file will conflict. Design your data layout so that concurrent writers touch different objects or partitions.

It is not a high-throughput ingestion bus. It versions batches. Streaming writes to a branch work, but each commit is a snapshot, and committing every second is the wrong shape. Commit on meaningful boundaries.

It is not an identity platform in Community. If you need per-team permissions, single sign-on and audit trails, you need the commercial editions or a design with one installation per team.

It is not a lightweight choice for a single practitioner's files. If your need is to version a few datasets alongside code in Git, DVC is lighter and keeps everything in the repository. If your need is experiment tracking and a model registry, that is MLflow, and lakeFS complements it by giving the data a stable, reproducible reference.

And since 1.87 it is not something you can host for others under the Community licence. A managed-data-platform business, or an internal team that would resell access to other legal entities, needs a commercial agreement or a different tool.

Choose lakeFS when you have several writers on a shared object store, need isolation for experiments and pipelines, need atomic publication of multi-file changes, want to gate data with checks before it reaches consumers, or need instant rollback and reproducibility by commit. That is a large set of real problems. It is not every problem.

Running lakeFS as a platform for other teams

The last piece is what turns an installation into a platform: the service you offer, how people get onto it, and how you keep it from becoming the thing everyone avoids.

Define the product. Write a one-page service description: what lakeFS is for in your organisation, which workloads it supports, the data path you recommend, the support hours, the retention window, the recovery objectives, and the contact. Include the licence constraint in plain language.

Offer a golden path. A team should get a working repository with a single request. Provide infrastructure code that creates the repository, the storage namespace with a correct prefix, branch protection on main, a standard set of action files for checks, and a GC rule. Use Terraform for the cloud pieces and a short script for the repository itself, for example:

lakectl repo create lakefs://ml-features-prod s3://acme-lakefs/ml-features-prod -d main
lakectl branch-protect add lakefs://ml-features-prod 'main'
lakectl gc set-config lakefs://ml-features-prod -f /platform/default-gc-rules.json

Standardise the client setup. Provide a base container image or a documented environment with lakectl installed at the pinned version, LAKECTL_SERVER_ENDPOINT_URL set, and credentials injected from the secret manager at runtime, never baked into the image. Provide a tested Spark configuration using lakeFSFS, a Python snippet using the high-level lakefs package, and a recipe for the S3 gateway for the tools that need it. On Kubernetes, ship the Helm values and remember that the chart's defaults, database.type: local and blockstore.type: local, are for development only.

Finally, region. If you run in the Gulf or Egypt, pick the bucket region and the database region deliberately, because data-residency rules can apply to both the data and the metadata. The setting installation.allow_inter_region_storage defaults to true, which permits repositories whose storage lives in a different region from the server, and you may want to disallow that. Keep the KV database and the object store in the same jurisdiction as your regulator expects, and note that commit metadata and branch names can themselves contain sensitive information.

The review checklist

Use this list for design documents and pull requests that touch your lakeFS platform.

• Version and licence. Is the version current and not on the blocklist (1.48.0, 1.74.1)? Has legal confirmed the Business Source License fits the use, including for any third party?
• Identity. Is the single-administrator model acceptable for this installation, or is Enterprise RBAC in the plan? Where is the administrator key stored, and when is it next rotated?
• Secrets. Are auth.encrypt.secret_key and blockstore.signing.secret_key in a secret manager and stable across releases? Are hook secrets injected through the actions.env.prefix mechanism?
• Configuration. Are deprecated auth.* keys removed? Is logging.format set to json and the audit level not below the main level? Is lakectl configured with an explicit endpoint?
• Data path. Which clients use the gateway, and is its bandwidth sized for them? Are native clients the default?
• Database. Does replica count times max_open_connections fit under the database limit? Are backups and point-in-time recovery enabled, and have they been restored in a drill?
• Storage. Is the IAM role minimal, with the required permissions on the prefix? Is versioning or replication enabled on the bucket? Is the namespace prefix unique and named for its owner?
• Retention. Is there a GC rule per repository, a schedule, and a first mark-only review? Is there a branch owner and expiry policy?
• Governance. Is main protected? Do gates have owners, timeouts and a bypass? Is the branch that supplies _lakefs_actions/ protected?
• Observability. Are the four dashboard rows in place, and do alerts exist for latency, errors, pool saturation and health?
• Recovery. Is there a written objective, and was the refs-dump and refs-restore path rehearsed on this version?
• Upgrade. Is there a written procedure with the migrate up boundary, and a rollback that restores a snapshot?
• Residency. Are the database, the bucket and the logs in the jurisdictions your regulator expects?

The complete picture

The thread through all of it is that lakeFS makes changes to data cheap, reviewable and reversible, and in exchange you take responsibility for the small number of things it cannot do for you: keeping the database healthy, keeping the secrets safe, running retention, and keeping the human workflow consistent. Teams that fail with lakeFS rarely fail on branching. They fail on an unmonitored database, an unowned branch that pinned a terabyte, a hook with no owner, or an upgrade that nobody read the release notes for.

Where the series leaves you

You can now design a lakeFS installation, defend its trade-offs, and run it through an upgrade like 1.87 without being surprised by it. Keep these habits: read every release note before you deploy, keep a version blocklist, rehearse recovery, protect main and the actions that guard it, and keep the administrator key as close to nobody as you can.

Where to go next depends on your gap. To get the tables on top of your versioned storage, read the Delta Lake guide. To run the garbage-collection job and the compute that reads lakeFS, read the Spark guide and, on a managed platform, the Databricks guide. To connect data versions to experiments and models, read the MLflow guide. To schedule the pipelines that branch, validate and merge, read the Airflow guide. To deploy and operate the platform itself, read the guides for Kubernetes, Helm, Terraform and Vault.

Sources

• lakeFS documentation
• Editions
• Architecture
• Internals
• Performance best practices
• Install and deploy
• Migrate from Community to Enterprise
• Upgrade
• Sizing guide
• Monitoring
• Backup and restore
• Troubleshooting
• Garbage collection
• Standalone garbage collection
• Protect branches
• Hooks
• Webhooks
• Security overview
• Pre-signed URLs
• Configuration reference
• lakectl reference
• S3 gateway reference
• Spark client
• Release notes
• lakeFS v1.87.0 release
• Licence (Business Source License 1.1)
• Security advisories
• Helm chart

Part three of three. At this level an interviewer is not checking whether you know the commands. They want to hear how you reason about a platform other teams depend on: where state lives, what fails first, what a change costs, and how you would explain a hard decision to someone who does not care about SSTables. This file is organised as system-design prompts, incident questions and leadership conversations. About fifty minutes. Facts are checked against lakeFS v1.87.0.

The two-minute answer

Start every senior interview with a compact, accurate description you can deliver without notes.

lakeFS is a Git-like version-control layer over an object store. Data stays in your bucket; lakeFS manages pointers and metadata, so a branch is a zero-copy metadata operation and a commit is an immutable snapshot of the whole repository. The server is a stateless Go binary serving the UI, the API and an S3 gateway. State lives in two places: committed metadata as immutable SSTables in the object store, and mutable state, meaning branch pointers, staged changes and credentials, in a key-value database such as PostgreSQL or DynamoDB. I run it for data isolation, atomic publication of multi-file changes, gated merges, and instant rollback.

Then volunteer the two facts that signal you keep current. First, since v1.87.0 Community is under the Business Source License and has a single administrator, so multi-user access, SSO and policies are Enterprise or Cloud. Second, a platform owner chooses the data path on purpose: the S3 gateway streams bytes through lakeFS, while native clients use pre-signed URLs and move data directly.

System-design prompt one: a shared data platform for forty teams

Prompt. "Design a versioned data platform for forty teams on one cloud. Some teams run Spark, some Python notebooks, some use R. Finance data must not be readable by marketing. We have one platform engineer."

A strong answer, in order.

Clarify before designing. Ask what needs isolating and why: is it a legal boundary, an audit requirement or a convenience? Ask about the scale (objects per repository, peak concurrent jobs), the clouds and regions, whether the company has a commercial lakeFS agreement, and whether any team is external to the legal entity. The last question matters because of the licence: the Community grant covers production use of the unmodified software for your organisation's own internal business purposes, and does not cover hosting for third parties.

Surface the constraint that shapes everything. Finance and marketing separation is a permissions requirement. Community v1.87 has one administrator and one credential, so it cannot express it. The options are an Enterprise or Cloud installation with RBAC, or separate Community installations per sensitivity domain with separate buckets, databases and administrator keys. With one platform engineer, a small number of installations (for example one for regulated data and one for general analytics) is realistic, and a per-team installation is not unless the deployment is fully automated.

Draw the architecture. Load balancer, several small stateless lakeFS servers, a managed PostgreSQL or DynamoDB for the key-value store, and buckets with one prefix per repository as the storage namespace. Terminate TLS at the load balancer, use /_health for checks, and scrape /metrics. Explain that the servers are replaceable and the database and bucket are where recovery effort goes.

Choose the data path. Native clients by default: Spark through lakeFSFS, Python through the lakefs package, lakectl for operators. The S3 gateway for R and other tools that only speak S3, sized for bandwidth. Explain why: the native path exchanges about a kilobyte of metadata per request with lakeFS and sends data directly, so the platform scales with requests, not bytes.

Design the tenancy. Repository per team and domain, with a naming convention that encodes owner and environment (repository names are three to sixty-three lowercase letters, numbers and hyphens), and never-reused namespace prefixes, because lakeFS does not delete data when a repository is deleted and reuse collides with leftover _lakefs data.

Add governance. Branch protection on main everywhere. A standard set of action files shipped by the platform: a pre-merge schema and quality check via webhook with an on-call owner, and a post-merge notification. Protect the branch that supplies the actions.

Close with cost and recovery. A GC rule per repository (default retention fourteen days is a reasonable starting point to propose), a scheduled GC run with a mark-only review the first time, branch expiry policy, database snapshots, bucket versioning, and a rehearsed refs-dump and refs-restore path.

What separates a strong answer from a weak one. A weak answer draws boxes and says "lakeFS handles permissions". A strong answer finds the identity constraint in the first two minutes, names the decision it forces, and states what it would cost to revisit. A weak answer forgets garbage collection entirely and discovers the storage bill in the last minute of the interview. A strong answer mentions it unprompted as the cost control.

System-design prompt two: reproducible ML training data

Prompt. "Our regulator asks which exact data trained the credit model deployed on 3 March. Design it so we can answer in minutes and retrain the same model."

A strong answer. The reference you want is a commit ID, because a commit is an immutable snapshot of the whole repository, including every object and its metadata. A branch name moves, so it is not a reference; a tag is an immutable name for a commit, and is readable. Design:

1. Training reads from lakefs://credit/<commit-id>/features/ or from a tag such as train-2026-03-01, never from main.
2. The training job records the commit ID in its experiment tracker run, for example as a parameter in MLflow, together with the Git commit of the training code. lakectl local and Git can tie a data commit to a code commit.
3. The pipeline that produces the dataset writes to a branch, validates, then merges into main with a pre-merge hook check. Each merge commit carries metadata (--meta k=v) such as the pipeline run ID and source system versions.
4. Retention protects it. Objects reachable from any branch head are kept, but a pinned commit that is not at any head can be collected by garbage collection after the retention window, and a later read returns 410 Gone. So keep a branch or protected reference for regulated releases, set retention long enough for the regulator, and record that decision.
5. Audit: lakectl log lakefs://credit/main --since <RFC3339 time> and lakectl show commit answer "what changed", and the commit metadata answers "by which run".

Follow-up you should expect. "Who changed the training set?" An honest answer for Community v1.87: the audit trail shows the action but not the person, because every client shares the single administrator identity. If the regulator needs individual attribution, you need Enterprise with per-user identities, or you record the human author in commit metadata by policy and enforce it with a pre-commit hook. The second is weaker, and you should say so.

System-design prompt three: bad data must never reach dashboards

Prompt. "A bad load corrupted a finance table last quarter and it took two days to find. Design it so that cannot reach consumers."

A strong answer. Use the write-audit-publish pattern, which lakeFS makes cheap. The pipeline writes to an ephemeral branch created from main (zero-copy, instant), the quality checks run against that branch, and only a passing branch is merged. A failing branch is deleted or left for investigation, and main was never touched. Enforce it rather than relying on goodwill: protect main so that direct writes fail with cannot write to protected branch, and add a pre-merge action whose webhook runs the checks, so the merge itself is gated.

Then discuss the failure semantics, because that is where depth shows. A pre-* hook failure aborts the operation, so the check service is now on the critical path: it needs an owner, a timeout (the default webhook timeout is one minute), a health check and a documented bypass. The airflow hook type returns once the trigger is accepted, so it cannot gate a merge on the DAG's result. Post-hooks cannot roll anything back. If a corrupted change does land, revert it with lakectl branch revert, adding -m 1 for a merge commit, and downstream readers pinned to earlier commits were never affected.

Mention the limits honestly: conflicts are per whole object, so concurrent pipelines writing the same file will conflict; and a check you did not think of will not catch the new kind of corruption, so the post-incident action is always a new check.

Incident questions

These are the prompts where they watch how you think under pressure. A good structure for every one is: state the impact, find the layer, stabilise, then fix and learn.

"After upgrading to 1.87 the server will not start." Read the first error. If it says lakeFS has no administrator of its own, and it holds repositories: run "lakefs superuser --user-name <name>"..., the new startup guard is doing its job: this installation has data but no administrator in the single-user store, typically because users lived in an ACL server or an external IAM. With the server stopped, run lakefs superuser --user-name admin --access-key-id <existing> --secret-access-key <existing> to keep the key pair your clients already hold, or omit the flags to get new credentials and redeploy clients. Then explain the root cause: the release removed pluggable IAM and the ACL reference server, and the release note said to migrate before upgrading. The lesson is that this upgrade needed a written plan, a snapshot and a staging rehearsal.

If the failure mentions cannot migrate existing user to basic auth mode, several users existed and the server cannot choose: pick the user whose keys matter and run the same command. Also say what you would do if the upgrade had to be undone: stop, restore the database snapshot taken before any migrate up, redeploy the previous version. Emphasise that the older binary cannot run against a migrated database.

"Every merge to main has been failing for twenty minutes." Check the action runs: lakectl actions runs list lakefs://repo --branch main, then describe on the failing run. A pre-merge webhook to an unhealthy service aborts every merge. Stabilise: if the check service cannot be fixed quickly, use the documented bypass, which might be disabling that one action file through a protected, reviewed change, or as a last resort actions.enabled: false on the server, which affects all repositories and should be reversed and recorded. Fix the dependency, then write the post-incident changes: a health check and alert on the webhook, a shorter timeout, and a runbook for the bypass.

"Readers of last year's training set get 410 Gone." Garbage collection removed objects the old commit referenced, because it was not reachable from a branch head and was past retention. Confirm with lakectl gc get-config, and with the GC mark report under the namespace's _lakefs/retention/gc/unified/ path. You cannot recover through lakeFS; recover from bucket-level object versioning if you have it. The fix is process: protect regulated commits with a reference that keeps them reachable, lengthen retention for that repository, and always run the first GC in mark-only mode with a human review.

Trade-off questions and how to argue them

Community or Enterprise? Decide on capabilities, not on price. Community v1.87 gives you one administrator, no SSO, no per-repository permissions, no audit identities, Spark-based garbage collection, and synchronous commits and merges. If none of those block your use case, Community under its licence is a good fit. If you need per-team permissions, SSO, SCIM, audit, multi-tenancy, standalone GC or asynchronous merge, you need Enterprise or Cloud, or multiple Community installations. Explain that the cost of the commercial edition should be compared with the cost of the engineering time you would spend working around the single identity, and with the risk of a shared key.

Gateway or native clients? Native whenever the tool supports it, because the data flow bypasses lakeFS and scales with the object store. Gateway where the tool can only speak S3, accepting that bandwidth and instance count now scale with data volume. The gateway also supports only a subset of S3: no server-side encryption requests, no SelectObject, no object tagging, and / as the only listing delimiter.

PostgreSQL or DynamoDB for the KV store? PostgreSQL is familiar, cheap at moderate scale and portable, with capacity planning around connections, shared_buffers and CPU. DynamoDB removes server management and scales on demand but can spike in cost under bursts and ties you to AWS. Pick what your team can operate and back up confidently. CosmosDB is the equivalent on Azure. The wrong answer is a large single PostgreSQL that is shared with other workloads and has no backup drill.

One installation or many? One installation reduces operational work and makes cross-team sharing easy, but needs RBAC to be safe, which means Enterprise. Many installations give hard isolation and blast-radius limits at the cost of repetition, which automation reduces. A hybrid by sensitivity domain is the usual landing place.

lakeFS or Delta Lake or Iceberg? They are not alternatives. Table formats give ACID transactions and schema evolution inside a table; lakeFS gives branching and atomic multi-object, multi-table commits across a repository, and works with any file format. Together they are common: lakeFS branches a repository that holds Delta tables (see Delta Lake).

lakeFS or DVC? DVC tracks data pointers in Git for a project and is light for a small team on a remote. lakeFS is a server for many concurrent writers on a shared store, with isolation, hooks and scale. Choose by the number of writers and the governance you need.

Arguing a trade-off to leadership

The skill here is translation. Leadership does not need the Merkle tree; they need the decision, the risk, the cost and the alternative, in that order. Four conversations come up repeatedly.

"Why are we paying for the commercial edition when the open-source one is free?" Do not say Community is inadequate. Say what the free edition cannot do for this company: it has one shared administrator identity, so we cannot tell who changed data, cannot give finance and marketing different access, and cannot revoke one person's access without rotating everyone's. Name the incident the shared key would cause (a leaked credential is a full compromise), and the audit finding it would produce. Put a number on the alternative: the engineering weeks to run separate installations per domain, plus their recurring operations. Offer the decision rule: if the regulated data is out of scope, we stay on Community and accept the risk in writing.

"Why did the upgrade to 1.87 need three weeks?" It was not a version bump. The release changed the licence, removed the authentication plug-in we relied on, and added a startup guard that refuses to start without an administrator. Each needed a decision (legal review of the licence terms, a choice between Enterprise and a single administrator, and a migration of client keys), and each had a failure mode that would have meant an outage. The three weeks bought a rehearsed rollback and no unplanned downtime. Show the before-and-after risk, not the technical detail.

"Why is storage growing when nobody added data?" Explain retained versions in one sentence: branching copies nothing, but every object ever written stays until garbage collection removes it, and an abandoned branch keeps what it references alive. Then show the plan: a retention policy, a scheduled GC run, a branch expiry rule, and the expected reclaimed storage from a mark-only report, which gives you a real number. Leadership responds to a projected saving with a date.

"Can we sell access to this platform to our customers?" Stop and involve legal. Under the Business Source License the free grant covers production use for your own internal business purposes and does not allow hosting it for or offering it to third parties, paid or unpaid. A customer-facing product built on it needs a commercial agreement with the licensor or a different technology. It is far better to say this in the planning meeting than in a licence review after launch.

A general pattern for all four: lead with the decision, give two options with cost and risk for each, recommend one, and say what evidence would change your recommendation.

Sixty-second self-test

Answer aloud, then check against the sections above.

1. Name the two places lakeFS keeps state and say what each failure costs.
2. Why do native clients scale better than the gateway?
3. What exactly changed about identity in v1.87.0, and what do you do when the server refuses to start?
4. Which licence covers Community now, and what may you not do with it?
5. What does a pre-merge hook failure do, and what about a post-merge failure?
6. Why can a read return 410 Gone, and how do you stop it happening to a regulated dataset?
7. Give the two commands that rebuild a repository from a reference dump.
8. Why does replica count matter for PostgreSQL?
9. Name two things lakeFS is not for.
10. What is your rollback boundary for an upgrade?

If you can answer all ten in a minute each, you are ready for the senior loop. If you stumble on three or more, go back to the architecture, security and upgrade sections of the detailed guide and rehearse them with a whiteboard.

Part three of three. At this level the problems are operational: a database that ran fine for a year and then saturated, a branch nobody owns that pins a terabyte, a hook that became a single point of failure, an upgrade that changed the licence and the authentication model in one release. Start with the error table, then the platform habits, the checklists for reviews, and the playbook recipes. Facts are checked against lakeFS v1.87.0.

Common errors at this level

Cumulative: everything from Beginner and Mid still applies. These cause incidents rather than failed commands.

Each of these has a common root: a change was treated as a version bump when it was a design change, or a safeguard was assumed to exist when nobody had built it. The habits below exist to catch that earlier.

Production traps that only appear at scale

The connection-pool multiplication

Each lakeFS server keeps up to database.postgres.max_open_connections connections, 25 by default. With three replicas that is 75 and nobody notices. With twenty replicas during a busy period it is 500 against a database configured for 200, and the failure looks like random API timeouts, not a clear error. Compute the product, leave headroom for migrations, admin tools and failover, and put a pooler in front when the arithmetic fails. Chart pool utilisation so you see it approach the ceiling in a graph and not in an incident.

The heavy reader on the gateway

A team points a Spark cluster of two hundred executors at the S3 gateway because the configuration was three lines. Every byte now flows through the lakeFS fleet, and your servers, which were sized for metadata, saturate their network. The symptom is high gateway latency with modest request counts. The fix is to move the job to lakeFSFS or another native path, which sends data directly to the object store using pre-signed URLs. Make this part of onboarding: the gateway is for tools that cannot do anything else.

The branch that pins a terabyte

An experiment branch rewrites a large table, commits, and is forgotten. Nothing is visibly wrong, because the branch is small in the UI and zero-copy made it free to create, but every object in its head is reachable and therefore never collected. Six months later the bucket has doubled. The habit is branch ownership and expiry: name branches with an owner, report branches older than policy weekly, and delete them. An object in any branch head is never collected, so deletion is what releases it.

The first GC run

Running garbage collection for the first time on a repository with years of history is where people discover the retention policy they really have. Do not run mark and sweep together the first time. Run mark only with spark.hadoop.lakefs.gc.do_sweep=false, read the report under <namespace>/_lakefs/retention/gc/unified/<MARK_ID>/, have the data owners review it, and only then sweep with lakefs.gc.do_mark=false and lakefs.gc.mark_id=<ID>. Before you start, check the platform constraints: GC does not support the local blockstore, Azure needs blob soft-delete disabled, and Google Cloud needs the service account to hold the Storage Object User role.

The imported data with a lifecycle rule

You imported from a source prefix and left a bucket lifecycle rule that expires objects after ninety days. lakeFS GC never touches imported objects, so you feel safe, but the bucket rule deletes them anyway and commits that reference them now point at nothing. Anything you import needs an explicit owner for the source prefix and its lifecycle configuration.

The hook that became infrastructure

A webhook added for a quick quality check is now in the path of every merge to main in forty repositories. It has no on-call, no alert and a one-minute default timeout, so when it is slow every engineer waits a minute per merge and when it is down nobody can publish data. Treat a synchronous gate as a production dependency from the first day: an owner, a health check, a latency budget, and a written bypass.

The action file as a code-execution path

Action files live in the repository under _lakefs_actions/. Anyone who can write to a branch can propose a Lua script or a webhook whose headers carry your server's environment secrets. If the branch that supplies actions is not protected and reviewed, a contributor to a data repository can cause the lakeFS server to run their code. Protect it, review changes to that directory, and keep actions.lua.net_http_enabled at its default of false unless a hook needs it.

The lakectl fleet drift

Old lakectl versions sit in CI images and laptops for years. A new lakectl talking to an old server, or the reverse, produces surprising errors: 1.74.1 introduced the no-default-endpoint change and was followed by 1.74.2, which fixed a panic when a new lakectl talks to an older server. An Enterprise server called from a Community lakectl prints a warning. Pin lakectl in the base image, rebuild the image on server upgrades, and make the endpoint explicit.

Platform-ownership habits

Read every release note, and write down what you decided

The 1.87 release is the argument for this habit. A release that changes the licence, removes authentication plug-ins, deprecates a family of configuration keys, adds a startup guard and changes the client in the same version is not something you learn about from a failed deploy. Subscribe to the releases, read the notes for every version you will cross, and keep a short decision record for each: what changed, what it means for us, what we did. Six months later, that file answers the audit question and the new-hire question.

Put the licence on the agenda

Since 1.87 Community is under the Business Source License 1.1 with an Additional Use Grant for internal business purposes and no hosting for third parties. Treat it like any dependency with a licence: record the decision, keep the licence file with the deployed artefact, and ask legal before a use that could be outside the grant, such as a platform that serves customers or another legal entity. Do not rely on the FAQ page that still says Apache 2.0; the current editions page and the licence file are the authority.

Decide the identity model once, in writing

Community has one administrator. That is a fine answer for a small trusted team and a poor one for a shared platform. Write down which you are running, who holds the key, how it is rotated, and the conditions under which you would move to Enterprise or split installations. The habit that prevents incidents is making the single identity a known, owned risk rather than an accident.

Automate the golden path

The platform is what a new team experiences in its first hour. Provide infrastructure code that creates a repository on a correctly prefixed namespace, turns on branch protection for main, installs the standard action files and sets a GC rule. A short script or Terraform module is enough. Teams that hand-build repositories produce fifty variations you cannot support. See the Terraform guide for the infrastructure side.

# Platform golden path for a new team repository
REPO=ml-features-prod
lakectl repo create "lakefs://$REPO" "s3://acme-lakefs/$REPO" -d main
lakectl branch-protect add "lakefs://$REPO" 'main'
lakectl gc set-config "lakefs://$REPO" -f /platform/default-gc-rules.json
lakectl branch-protect list "lakefs://$REPO"

Make recovery a practised skill

Run the restore drill twice a year and after every major upgrade: create a clean installation from code, restore the database snapshot, point it at a copy of the bucket, and verify a known tag and object. Include lakectl refs-dump and refs-restore on a bare repository in the drill, and pin the lakectl version you use, because those commands are listed as undocumented plumbing and could change. Record the time the drill took; that is your real recovery time, whatever the document says.

Measure upgrade lag

The best single indicator of platform health is how many versions behind the latest release you are. Recent security fixes (the gateway replay issue in 1.75, the local-adapter path traversal in 1.77, the dependency fix in 1.82, the gateway listing fix in 1.86) are each a reason to be current. Report the lag monthly. If it grows, you will meet the next large change as a forced migration, as 1.87 was for installations that had fallen behind and depended on the plug-in.

Keep the exit open

Periodically export the current state of critical repositories to plain paths in a separate bucket, and keep the refs-dump manifests alongside. The point is not that you intend to leave; it is that depending on a tool responsibly means you know how you would. The data stays in your bucket in a documented layout, which already makes the exit easier than for most systems.

Write the runbooks before the incidents

Keep a short runbook per failure: API slow, merges failing, 410 Gone, leaked key, server will not start after upgrade, database lost. Each should list the first three commands or metrics, the decision points and the owner. Review them after each incident. See the Prometheus and Grafana guides for the dashboards that feed them.

Tell users the rules they live under

Publish retention, branch expiry, the recommended data path, the commit-message and metadata conventions, and the support route. Users who know that GC can produce 410 Gone on old unpinned commits pin the references they care about. Users who do not know it file incidents.

Review checklists

Pull request checklist: changes to the platform repository

• Does it change a version? Is the new version current, not on the blocklist, and have the release notes for every crossed version been read and summarised in the description?
• Does it touch auth.* keys? Are deprecated keys removed, and are the live keys (auth.encrypt.secret_key, auth.cache.*, auth.login_duration, auth.logout_redirect_url) unchanged in value?
• Does it template a secret from a random source? It must not; secrets come from the secret manager.
• Does it change blockstore.*? Was the type left alone, the region set explicitly, and blockstore.signing.secret_key kept?
• Does it change database.*? Has replica count times max_open_connections been checked against the database limit?
• Does it add or change an action file? Is it validated with lakectl actions validate, is it on a protected branch, does it have a timeout, and does a bypass exist?
• Does it create repositories? Does the namespace prefix follow the convention and avoid any prefix previously used?
• Does it change logging? Is logging.format still json, and is the audit level not below the main level?
• Does it change ingress? Is proxy-body-size set high enough, and is TLS still terminated at the load balancer?
• Does it change Helm values? Are the development defaults (database.type: local, blockstore.type: local, useDevPostgres) absent from production values? See the Helm guide.

Design document checklist: a new use of lakeFS

• Purpose. What problem does versioning solve here: isolation, atomic publish, gating, rollback, reproducibility? If none, why lakeFS?
• Identity. Who needs access, with what permissions, and does the installation's edition support it?
• Licence. Is the use internal to the organisation? Does any third party touch the installation?
• Data path. Which clients read and write, through which path, and what is the expected throughput?
• Layout. Do concurrent writers touch different objects or partitions, given that conflicts are per whole object?
• Retention. What is the retention window, which commits must outlive it, and what protects them?
• Governance. Which branches are protected, which checks gate merges, who owns the checks, and what is the bypass?
• Recovery. Which backups cover data and pointers, and when was the restore last rehearsed?
• Observability. Which dashboards and alerts does this workload need, and who responds?
• Residency. Where do the bucket, database and logs live, and does that satisfy the regulator?
• Exit. How would we export this data in a form other tools read?

Review checklist: a data pipeline that uses lakeFS

• Does the pipeline write to an ephemeral branch and merge only on success?
• Is the branch name unique per run, and is the branch deleted or expired afterwards?
• Are commits made with messages and metadata that record the run and the code version?
• Does failure handling reset or delete the branch without touching main?
• Do readers reference a tag or commit ID when reproducibility matters, not a branch?
• Does the pipeline use branch.transact() or an equivalent, so a multi-step change is atomic?
• Is retry behaviour bounded, given that lakectl retries four times by default and a 429 means back off?

Operational recipes

Health, identity and version checks

curl -i https://lakefs.example.com/_health
lakectl identity
lakectl doctor
lakefs --version ; lakectl --version

The health endpoint returns 200 when healthy and is the endpoint for load balancer checks. lakectl doctor checks configuration, credentials and connectivity. Put all four into your post-deploy verification.

A scripted post-upgrade smoke test

set -euo pipefail
REPO=smoke-test
lakectl repo create "lakefs://$REPO" "s3://acme-lakefs/$REPO" -d main
echo "hello" | lakectl fs upload "lakefs://$REPO/main/hello.txt" -s -
lakectl commit "lakefs://$REPO/main" -m "smoke commit"
lakectl branch create "lakefs://$REPO/feature" -s "lakefs://$REPO/main"
lakectl fs cat "lakefs://$REPO/feature/hello.txt"
lakectl branch delete "lakefs://$REPO/feature" -y
lakectl repo delete "lakefs://$REPO"

Use a prefix reserved for smoke tests and never reused, because repository deletion does not remove the data and the next run would hit storage namespace already in use.

Configure retention

cat > rules.json <<'EOF'
{
  "default_retention_days": 14,
  "branches": [
    { "branch_id": "main", "retention_days": 21 }
  ]
}
EOF
lakectl gc set-config lakefs://my-repo -f rules.json
lakectl gc get-config lakefs://my-repo

Check what a hook did

lakectl actions runs list lakefs://my-repo --branch main
lakectl actions runs describe lakefs://my-repo <run-id>
lakectl actions validate ./_lakefs_actions/pre-merge-quality.yaml

Revert bad data on main

lakectl log lakefs://my-repo/main --amount 5
lakectl branch revert lakefs://my-repo/main <commit-id> -y
# For a merge commit, name the parent to keep (1 is the destination side):
lakectl branch revert lakefs://my-repo/main <merge-commit-id> -m 1 -y

Save and restore references

lakectl refs-dump lakefs://my-repo > refs_manifest.json
lakectl repo create-bare lakefs://my-repo-restored s3://restored-bucket/my-repo
lakectl refs-restore lakefs://my-repo-restored --manifest refs_manifest.json

Collect diagnostics safely

lakefs flare

The diagnostic bundle redacts secrets, but read it before sharing it outside your organisation.

Query the latency that matters

sum by (operation) (histogram_quantile(0.99, rate(api_request_duration_seconds_bucket[1m])))

Build alerts from this and from kv_request_duration_seconds and the pgxpool_* family; see the Prometheus guide.

A final checklist for the platform owner

• I know where every piece of state lives and how each is backed up.
• The administrator key has one owner, one location and a rotation date.
• The encryption and signing secrets are stable across releases and recoverable.
• Deprecated auth.* keys are gone from the configuration.
• Replica count times pool size fits the database, and I chart the pool.
• Native clients are the default; the gateway is sized for what uses it.
• Every repository has a GC rule, every stale branch has an owner, and the first GC ran in two phases.
• main is protected, gates have owners and bypasses, and the actions directory is protected.
• Dashboards, alerts and runbooks exist and were tested in a game day.
• The restore drill ran this half-year, and I know the time it took.
• I am within two releases of current, and I keep a blocklist.
• Legal has recorded that the Business Source License fits our use.
• I know the exit, and I have tested that I can export.