Problem with /tmp file #1793

Activity

1. stefanprodan commented on May 7, 2025

   

   stefanprodan

   on May 7, 2025

   Member

   More actions

   Fluxcd should probably clean-up after itself?

   It does so probably something else is blocking it on the node. Look at source-controller logs for any errors.
2. monsad commented on May 12, 2025

   

   monsad

   on May 12, 2025

   Author

   More actions

   No, I checked source-controller logs and I found only 'info' level logs, any errors. So I looks that /tmp is not clean up
3. acondrat commented on May 22, 2025

   

   acondrat

   on May 22, 2025

   Last edited by acondrat

   More actions

   We noticed the same issue. We run the source controller in kubernetes with the /tmp mounted on an emtyDir volume (see spec in this chart).

   As far as I can tell every time we run flux reconcile source git mygitrepo a new folder is created in /tmp that looks like /tmp/gitrepository-flux-system-mygitrepo-someid. This folder exists for a brief period (I assume some checksums are compared) and then it gets removed.

   The problem is that this removal doesn't always happen. I can't consistently reproduce it but I see some leftover folders that were never deleted. In our case it eventually leads to increased memory usage and the container gets OOMKilled.

   I checked the logs but all I have is "info" level messages and no indication of a problem whatsoever.

   $ flux  version 
   flux: v2.5.1
   distribution: flux-2.5.1
   helm-controller: v1.2.0
   image-automation-controller: v0.40.0
   image-reflector-controller: v0.34.0
   kustomize-controller: v1.5.1
   notification-controller: v1.5.0
   source-controller: v1.5.0
   
4. stefanprodan commented on May 24, 2025

   

   stefanprodan

   on May 24, 2025

   Member

   More actions

   I can't explain how would the tmp cleanup fail without having the error logged:

   source-controller/internal/controller/gitrepository_controller.go

   Lines 279 to 293 in 4aa31dc

   	// Create temp dir for Git clone
   	tmpDir, err := util.TempDirForObj("", obj)
   	if err != nil {
   	e := serror.NewGeneric(
   	fmt.Errorf("failed to create temporary working directory: %w", err),
   	sourcev1.DirCreationFailedReason,
   	)
   	conditions.MarkTrue(obj, sourcev1.StorageOperationFailedCondition, e.Reason, "%s", e)
   	return sreconcile.ResultEmpty, e
   	}
   	defer func() {
   	if err = os.RemoveAll(tmpDir); err != nil {
   	ctrl.LoggerFrom(ctx).Error(err, "failed to remove temporary working directory")
   	}
   	}()

   No matter what happens during the reconciliation, we remove the tmp dir and if it fails, we log an error.
5. acondrat commented on Jun 3, 2025

   

   acondrat

   on Jun 3, 2025

   Last edited by acondrat

   More actions

   I suspect the cleanup fails if the container gets OOMKilled in the middle of it. After this happens a few times and more garbage is accumulated in tmp the container constantly gets OOMKilled on startup. The workaround for us was to delete the pod so both /tmp and /data (emptyDir) are wiped.
   We also allocated more memory and it no longer seems to happen.
6. andrewb3000 commented on Oct 1, 2026

   

   andrewb3000

   on Oct 1, 2026

   More actions

   We hit this on edge clusters running v1.9. The cleanup is a defer, so it never runs when the process exits abruptly: on a lost leader lease (os.Exit(1)) or an OOM kill. The emptyDir survives the container restart, so each crash leaves the in-flight clones behind. With a few hundred restarts a month, /tmp reached 7.5 GB on one of our nodes and pods around it were evicted for disk pressure.

   This is the same leak kustomize-controller had in fluxcd/kustomize-controller#1723, fixed there by purging its tmp dirs at startup (fluxcd/kustomize-controller#1729). I've opened #2176 doing the equivalent here.

   👍React with 👍1Reacted by Dongfang Qu