boltfs

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 12, 2025 License: MIT Imports: 17 Imported by: 0

README

BoltFS - A Complete Filesystem Implementation for BoltDB

Go Reference Go Report Card CI License

BoltFS provides a full featured filesystem on top of a boltdb database. This package implements most of the filesystem functions from the os standard library package, even including support for symbolic links.

Features

  • Compatible with the abstract filesystem interface absfs.SymlinkFileSystem
  • Support for hard and soft linking
  • Walk method like filepath.Walk
  • Thread-safe in-memory inode cache for improved performance
  • Snapshot support using BoltDB's MVCC for point-in-time filesystem views
  • External content storage for storing file content outside BoltDB (filesystem, S3, etc.)
  • Extensive tests with >90% coverage
Inode Cache

The inode cache provides significant performance improvements by caching frequently accessed inodes in memory:

fs, _ := boltfs.Open("myfs.db", "")
defer fs.Close()

// Get cache statistics
stats := fs.CacheStats()
fmt.Printf("Hit rate: %.2f%%\n", stats.HitRate())

// Adjust cache size (default: 1000)
fs.SetCacheSize(5000)

// Flush cache if needed
fs.FlushCache()
Snapshots

Create read-only point-in-time views of the filesystem:

// Create a snapshot
snapshot, _ := fs.CreateSnapshot("backup-2024")
defer snapshot.Release()

// Read files from snapshot
data, _ := snapshot.ReadFile("/config.json")

// Walk snapshot filesystem
snapshot.Walk("/", func(path string, info os.FileInfo, err error) error {
    fmt.Println(path)
    return nil
})

// Restore files from snapshot
snapshot.CopyToFS("/config.json", "/config-restored.json")

// Use snapshot manager for multiple snapshots
sm := fs.NewSnapshotManager()
sm.Create("daily-backup")
snap, _ := sm.Get("daily-backup")
External Content Storage

Store file content in any absfs.FileSystem implementation while keeping metadata in BoltDB:

import (
    "github.com/absfs/boltfs"
    "github.com/absfs/memfs"  // or any absfs implementation
)

// Use default BoltDB storage (backward compatible)
fs, _ := boltfs.Open("myfs.db", "")
defer fs.Close()

// Use memfs for file content (in-memory storage)
contentFS, _ := memfs.NewFS()
fs, _ := boltfs.OpenWithContentFS("myfs.db", "", contentFS)
defer fs.Close()

// Or set content filesystem on existing filesystem
contentFS, _ := memfs.NewFS()
fs.SetContentFS(contentFS)

// Use with existing bolt.DB
db, _ := bolt.Open("myfs.db", 0644, nil)
contentFS, _ := memfs.NewFS()
fs, _ := boltfs.NewFSWithContentFS(db, "", contentFS)

Custom Storage Backends:

Any absfs.FileSystem implementation can be used:

// Example: osfs for local filesystem storage
type OSFileSystem struct {
    basePath string
}

func (fs *OSFileSystem) OpenFile(name string, flag int, perm os.FileMode) (absfs.File, error) {
    // Implement absfs.FileSystem interface
    // Store files in basePath + name
}
// ... implement other absfs.FileSystem methods

// Use custom filesystem
osfs := &OSFileSystem{basePath: "/data/content"}
fs, _ := boltfs.OpenWithContentFS("myfs.db", "", osfs)

Benefits:

  • Idiomatic design: Uses standard absfs.FileSystem interface
  • Composable: Chain/layer multiple absfs implementations
  • Rich ecosystem: Use any existing absfs implementation (memfs, osfs, s3fs, etc.)
  • Backward compatible: Defaults to BoltDB storage
  • Automatic cleanup: Files deleted from external storage when removed

Coming soon

  • Improved test coverage
  • FastWalk high performance walker (non sorted, os.FileMode only)

License

MIT license. See LICENSE file for more information.

Documentation

Overview

Package boltfs provides a complete file system implementation for boltdb. This implementation includes support for symbolic links and includes a file system walker that works just like `filepath.Walk`.

BoltFS now includes advanced features:

  • Thread-safe in-memory inode cache for improved performance
  • Snapshot support using BoltDB's MVCC for point-in-time views

The inode cache significantly improves read performance by caching frequently accessed inodes in memory with configurable size and LRU eviction.

Snapshots provide read-only, consistent views of the filesystem at a point in time without blocking writes, implemented using BoltDB's read-only transactions.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type CacheStats 

type CacheStats struct {
	Size    int    // Current number of cached entries
	MaxSize int    // Maximum cache size
	Hits    uint64 // Number of cache hits
	Misses  uint64 // Number of cache misses
	Enabled bool   // Whether the cache is enabled
}

CacheStats contains statistics about cache performance.

func (CacheStats) HitRate 

func (s CacheStats) HitRate() float64

HitRate returns the cache hit rate as a percentage.

type File 

type File struct {
	// contains filtered or unexported fields
}

File implements the absfs.File interface, providing a file interface for boltdb

func (*File) Close 

func (f *File) Close() error

Close closes the File, rendering it unusable for I/O. On files that support SetDeadline, any pending I/O operations will be canceled and return immediately with an error.

func (*File) Name 

func (f *File) Name() string

Name returns the name of the file as presented to Open.

func (*File) Read 

func (f *File) Read(p []byte) (int, error)

Read reads up to len(b) bytes from the File. It returns the number of bytes read and any error encountered. At end of file, Read returns 0, io.EOF.

func (*File) ReadAt 

func (f *File) ReadAt(b []byte, off int64) (n int, err error)

ReadAt reads len(b) bytes from the File starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF.

func (*File) ReadDir 

func (f *File) ReadDir(n int) ([]fs.DirEntry, error)

ReadDir reads the contents of the directory and returns a slice of up to n DirEntry values in directory order. This is the modern Go 1.16+ equivalent of Readdir that returns lightweight DirEntry values instead of full FileInfo.

If n > 0, ReadDir returns at most n entries. In this case, if ReadDir returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF.

If n <= 0, ReadDir returns all entries from the directory in a single slice. In this case, if ReadDir succeeds (reads all the way to the end of the directory), it returns the slice and a nil error.

func (*File) Readdir 

func (f *File) Readdir(n int) ([]os.FileInfo, error)

Readdir reads the contents of the directory associated with file and returns a slice of up to n FileInfo values, as would be returned by Lstat, in directory order. Subsequent calls on the same file will yield further FileInfos.

If n > 0, Readdir returns at most n FileInfo structures. In this case, if Readdir returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF.

If n <= 0, Readdir returns all the FileInfo from the directory in a single 

slice. In this case, if Readdir succeeds (reads all the way to the end of

the directory), it returns the slice and a nil error. If it encounters an error before the end of the directory, Readdir returns the FileInfo read until that point and a non-nil error.

func (*File) Readdirnames 

func (f *File) Readdirnames(n int) ([]string, error)

Readdirnames reads and returns a slice of names from the directory f.

If n > 0, Readdirnames returns at most n names. In this case, if Readdirnames returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF.

If n <= 0, Readdirnames returns all the names from the directory in a single slice. In this case, if Readdirnames succeeds (reads all the way to the end of the directory), it returns the slice and a nil error. If it encounters an error before the end of the directory, Readdirnames returns the names read until that point and a non-nil error.

func (*File) Seek 

func (f *File) Seek(offset int64, whence int) (ret int64, err error)

Seek sets the offset for the next Read or Write on file to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. It returns the new offset and an error, if any. The behavior of Seek on a file opened with O_APPEND is not specified.

func (*File) Stat 

func (f *File) Stat() (os.FileInfo, error)

Stat returns the FileInfo structure describing file. If there is an error, it will be of type *PathError.

func (*File) Sync 

func (f *File) Sync() error

Sync does nothing because Write always completes before returning

func (*File) Truncate 

func (f *File) Truncate(size int64) error

Truncate changes the size of the named file. If the file is a symbolic link, it changes the size of the link's target. If there is an error, it will be of type *PathError.

func (*File) Write 

func (f *File) Write(p []byte) (int, error)

func (*File) WriteAt 

func (f *File) WriteAt(b []byte, off int64) (n int, err error)

WriteAt writes len(b) bytes to the File starting at byte offset off. It returns the number of bytes written and an error, if any. WriteAt returns a non-nil error when n != len(b).

func (*File) WriteString 

func (f *File) WriteString(s string) (n int, err error)

WriteString is like Write, but writes the contents of string s rather than a slice of bytes.

type FileSystem 

type FileSystem struct {
	// contains filtered or unexported fields
}

FileSystem implements absfs.FileSystem for the boltdb packages `github.com/coreos/bbolt`.

func NewFS 

func NewFS(db *bolt.DB, bucketpath string) (*FileSystem, error)

NewFS creates a new FileSystem pointer in the convention of other `absfs`, implementations. It takes a bolt.DB pointer, and a bucket name to use as the storage location for the file system buckets. If `bucket` is an empty string file system buckets are created as top level buckets.

func NewFSWithContentFS 

func NewFSWithContentFS(db *bolt.DB, bucketpath string, contentFS absfs.FileSystem) (*FileSystem, error)

NewFSWithContentFS creates a new FileSystem with an external content filesystem. This is useful for storing file content externally while keeping metadata in BoltDB.

func Open 

func Open(path, bucketpath string) (*FileSystem, error)

Open takes an absolute or relative path to a `boltdb` file and an optionl bucket name to store boltfs buckets. If `bucket` is an empty string file system buckets are created as top level buckets. If the bolt database already exists it will be loaded, otherwise a new database is created with with default configuration.

func OpenWithContentFS 

func OpenWithContentFS(path, bucketpath string, contentFS absfs.FileSystem) (*FileSystem, error)

OpenWithContentFS opens a BoltDB filesystem with an external content filesystem. This is useful for storing file content externally while keeping metadata in BoltDB.

func (*FileSystem) CacheStats 

func (fs *FileSystem) CacheStats() CacheStats

CacheStats returns statistics about the inode cache.

func (*FileSystem) Chdir 

func (fs *FileSystem) Chdir(name string) error

Chdir - changes the current directory to the absolute or relative path provided by `Chdir`

func (*FileSystem) Chmod 

func (fs *FileSystem) Chmod(name string, mode os.FileMode) error

Chmod changes the mode of the named file to mode.

func (*FileSystem) Chown 

func (fs *FileSystem) Chown(name string, uid, gid int) error

Chown changes the owner and group ids of the named file

func (*FileSystem) Chtimes 

func (fs *FileSystem) Chtimes(name string, atime time.Time, mtime time.Time) error

Chtimes changes the access and modification times of the named file

func (*FileSystem) Close 

func (fs *FileSystem) Close() error

Close waits for pending writes, then closes the database file.

func (*FileSystem) Copy 

func (fs *FileSystem) Copy(source, destination string) error

Copy is a convenience function that duplicates the `source` path to the `newpath`

func (*FileSystem) Create 

func (fs *FileSystem) Create(name string) (absfs.File, error)

Create is a convenience function that opens a file for reading and writing. If the file does not exist it is created, if it does then it is truncated.

func (*FileSystem) CreateSnapshot 

func (fs *FileSystem) CreateSnapshot(name string) (*Snapshot, error)

CreateSnapshot creates a new read-only snapshot of the filesystem. The snapshot must be released by calling Release() when done to free resources.

func (*FileSystem) FlushCache 

func (fs *FileSystem) FlushCache()

FlushCache removes all entries from the inode cache.

func (*FileSystem) Getwd 

func (fs *FileSystem) Getwd() (dir string, err error)

Getwd returns the current working directory, the error value is always `nil`.

func (*FileSystem) Lchown 

func (fs *FileSystem) Lchown(name string, uid, gid int) error

Lchown changes the numeric uid and gid of the named file. If the file is a symbolic link, it changes the uid and gid of the link itself. If there is an error, it will be of type *PathError.

On Windows, it always returns the syscall.EWINDOWS error, wrapped in *PathError.

func (*FileSystem) Lstat 

func (fs *FileSystem) Lstat(name string) (os.FileInfo, error)

Lstat returns a FileInfo describing the named file. If the file is a symbolic link, the returned FileInfo describes the symbolic link. Lstat makes no attempt to follow the link. If there is an error, it will be of type *PathError.

func (*FileSystem) Mkdir 

func (fs *FileSystem) Mkdir(name string, perm os.FileMode) error

Mkdir creates a new directory with the specified name and permission bits (before umask). If there is an error, it will be of type *PathError.

func (*FileSystem) MkdirAll 

func (fs *FileSystem) MkdirAll(name string, perm os.FileMode) error

MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. The permission bits perm (before umask) are used for all directories that MkdirAll creates. If path is already a directory, MkdirAll does nothing and returns nil.

func (*FileSystem) NewSnapshotManager 

func (fs *FileSystem) NewSnapshotManager() *SnapshotManager

NewSnapshotManager creates a new snapshot manager for the given filesystem.

func (*FileSystem) Open 

func (fs *FileSystem) Open(name string) (absfs.File, error)

Open is a convenience function that opens a file in read only mode.

func (*FileSystem) OpenFile 

func (fs *FileSystem) OpenFile(name string, flag int, perm os.FileMode) (absfs.File, error)

OpenFile is the generalized open call; most users will use Open or Create instead. It opens the named file with specified flag (O_RDONLY etc.) and perm (before umask), if applicable. If successful, methods on the returned File can be used for I/O. If there is an error, it will be of type *os.PathError.

func (*FileSystem) ReadDir 

func (filesystem *FileSystem) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir reads the named directory and returns a list of directory entries sorted by filename.

func (*FileSystem) ReadFile 

func (fs *FileSystem) ReadFile(name string) ([]byte, error)

ReadFile reads the named file and returns its contents. 

func (fs *FileSystem) Readlink(name string) (string, error)

Readlink returns the destination of the named symbolic link. If there is an error, it will be of type *PathError.

func (*FileSystem) Remove 

func (fs *FileSystem) Remove(name string) error

Remove removes the named file or (empty) directory. If there is an error, it will be of type *PathError.

func (*FileSystem) RemoveAll 

func (fs *FileSystem) RemoveAll(name string) error

RemoveAll removes path and any children it contains. It removes everything it can but returns the first error it encounters. If the path does not exist, RemoveAll returns nil (no error).

func (*FileSystem) Rename 

func (fs *FileSystem) Rename(oldpath, newpath string) error

Rename renames (moves) oldpath to newpath. If newpath already exists and is not a directory, Rename replaces it. OS-specific restrictions may apply when oldpath and newpath are in different directories. If there is an error, it will be of type *LinkError.

func (*FileSystem) SetCacheSize 

func (fs *FileSystem) SetCacheSize(size int)

SetCacheSize changes the maximum size of the inode cache. Setting size to 0 or negative disables the cache.

func (*FileSystem) SetContentFS 

func (fs *FileSystem) SetContentFS(contentFS absfs.FileSystem)

SetContentFS sets an external filesystem for storing file content. This allows file content to be stored in any absfs.FileSystem implementation (memfs, osfs, s3fs, etc.) instead of in BoltDB. This should be called before any file operations.

func (*FileSystem) SetTempdir 

func (fs *FileSystem) SetTempdir(tempdir string)

SetTempdir sets the path to a temporary directory, but does not create the actual directories. Silently ignores errors.

func (*FileSystem) SetUmask 

func (fs *FileSystem) SetUmask(umask os.FileMode)

SetUmask sets the current `umask` value. Silently ignores errors.

func (*FileSystem) Stat 

func (fs *FileSystem) Stat(name string) (os.FileInfo, error)

Stat returns the FileInfo structure describing file. If there is an error, it will be of type *os.PathError.

func (*FileSystem) Sub 

func (fs *FileSystem) Sub(dir string) (fs.FS, error)

Sub returns an fs.FS corresponding to the subtree rooted at dir. 

func (fs *FileSystem) Symlink(source, destination string) error

Symlink creates newname as a symbolic link to oldname. If there is an error, it will be of type *LinkError.

func (*FileSystem) TempDir 

func (fs *FileSystem) TempDir() string

TempDir returns the path to a temporary directory. Returns "/tmp" if an error occurs.

func (*FileSystem) Truncate 

func (fs *FileSystem) Truncate(name string, size int64) error

Truncate changes the size of the file. It does not change the I/O offset. If there is an error, it will be of type *os.PathError.

func (*FileSystem) Umask 

func (fs *FileSystem) Umask() os.FileMode

Umask returns the current `umask` value. A non zero `umask` will be masked with file and directory creation permissions. Returns 0755 if an error occurs.

type Snapshot 

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot represents a read-only point-in-time view of the filesystem. Snapshots are implemented using BoltDB's read-only transactions (MVCC), which provide consistent views without blocking writes to the main filesystem.

func (*Snapshot) Chdir 

func (s *Snapshot) Chdir(dir string) error

func (*Snapshot) Chmod 

func (s *Snapshot) Chmod(name string, mode os.FileMode) error

func (*Snapshot) Chown 

func (s *Snapshot) Chown(name string, uid, gid int) error

func (*Snapshot) Chtimes 

func (s *Snapshot) Chtimes(name string, atime, mtime time.Time) error

func (*Snapshot) CopyToFS 

func (s *Snapshot) CopyToFS(srcPath, dstPath string) error

CopyToFS copies a file or directory from the snapshot to the main filesystem.

func (*Snapshot) Create 

func (s *Snapshot) Create(name string) (absfs.File, error)

func (*Snapshot) Created 

func (s *Snapshot) Created() time.Time

Created returns the time when the snapshot was created.

func (*Snapshot) Getwd 

func (s *Snapshot) Getwd() (string, error)

func (*Snapshot) Mkdir 

func (s *Snapshot) Mkdir(name string, perm os.FileMode) error

func (*Snapshot) MkdirAll 

func (s *Snapshot) MkdirAll(path string, perm os.FileMode) error

func (*Snapshot) Name 

func (s *Snapshot) Name() string

Name returns the snapshot's name.

func (*Snapshot) Open 

func (s *Snapshot) Open(name string) (absfs.File, error)

func (*Snapshot) OpenFile 

func (s *Snapshot) OpenFile(name string, flag int, perm os.FileMode) (absfs.File, error)

func (*Snapshot) ReadDir 

func (s *Snapshot) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir reads the directory named by path in the snapshot and returns a list of directory entries.

func (*Snapshot) ReadFile 

func (s *Snapshot) ReadFile(name string) ([]byte, error)

ReadFile reads the entire file at the given path in the snapshot. 

func (s *Snapshot) Readlink(name string) (string, error)

Readlink reads the target of a symbolic link in the snapshot.

func (*Snapshot) Release 

func (s *Snapshot) Release() error

Release closes the snapshot and frees associated resources. The snapshot cannot be used after calling Release().

func (*Snapshot) Remove 

func (s *Snapshot) Remove(name string) error

func (*Snapshot) RemoveAll 

func (s *Snapshot) RemoveAll(path string) error

func (*Snapshot) Rename 

func (s *Snapshot) Rename(oldpath, newpath string) error

func (*Snapshot) Stat 

func (s *Snapshot) Stat(name string) (os.FileInfo, error)

Stat returns file information for the given path in the snapshot.

func (*Snapshot) Sub 

func (s *Snapshot) Sub(dir string) (fs.FS, error)

Sub returns an fs.FS corresponding to the subtree rooted at dir in the snapshot.

func (*Snapshot) TempDir 

func (s *Snapshot) TempDir() string

func (*Snapshot) Truncate 

func (s *Snapshot) Truncate(name string, size int64) error

type SnapshotManager 

type SnapshotManager struct {
	// contains filtered or unexported fields
}

SnapshotManager manages named snapshots for a filesystem.

func (*SnapshotManager) Create 

func (sm *SnapshotManager) Create(name string) (*Snapshot, error)

Create creates a new named snapshot.

func (*SnapshotManager) Delete 

func (sm *SnapshotManager) Delete(name string) error

Delete releases and removes a snapshot by name.

func (*SnapshotManager) Get 

func (sm *SnapshotManager) Get(name string) (*Snapshot, bool)

Get retrieves a snapshot by name.

func (*SnapshotManager) List 

func (sm *SnapshotManager) List() []string

List returns a list of all snapshot names.

func (*SnapshotManager) ReleaseAll 

func (sm *SnapshotManager) ReleaseAll() error

ReleaseAll releases all snapshots.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.