File Storage

Overview

Spiderly's file-storage model is per-property: every blob property on an entity declares its storage adapter via a StorageAttribute subclass. The source generator emits the upload pipeline (controller endpoint, validation, optimization hooks, save-time cleanup) and resolves the right IFileManager adapter from DI for that property only — there is no global storage registration.

Built-in Adapters

Spiderly ships three built-in storage classes and matching attributes. Anything else (Cloudinary, Azure Blob, Backblaze, on-prem MinIO, …) is implemented by the consumer (see Custom Adapters).

All three implement Spiderly.Shared.Interfaces.IFileManager.

Entity Configuration

Decorate a string property with a storage attribute. The attribute itself marks the property as a blob — there is no separate marker attribute.

Public CDN file

The column stores the full public URL. Ideal for images served directly from a CDN:

public class Brand : BusinessObject<int>
{
    [S3PublicStorage]
    [AcceptedFileTypes("image/jpeg", "image/png", "image/webp", "image/avif")]
    [MaxFileSize(2_000_000)]
    [StringLength(1000, MinimumLength = 1)]
    public string LogoUrl { get; set; }
}

Private S3 file (signed-URL access)

The column stores an opaque S3 key; access is mediated via signed URLs or a backend proxy. Use for personal data, compliance-sensitive uploads, etc.:

public class WarrantyRegistration : BusinessObject<long>
{
    [S3PrivateStorage]
    [AcceptedFileTypes("image/jpeg", "image/png", "application/pdf")]
    [MaxFileSize(10_000_000)]
    [StringLength(1000, MinimumLength = 1)]
    public string ReceiptImageUrl { get; set; }
}

Local disk (development)

public class User : BusinessObject<long>
{
    [DiskStorage]
    [AcceptedFileTypes("image/*")]
    [StringLength(1000, MinimumLength = 1)]
    public string ProfilePicture { get; set; }
}

File Validation Attributes

These attributes add both server-side and client-side validation. See the Validation page for details.

Example with all validation attributes

public class Brand : BusinessObject<int>
{
    [DisplayName]
    [Required]
    [StringLength(100, MinimumLength = 1)]
    public string Name { get; set; }

    [S3PublicStorage]
    [AcceptedFileTypes("image/*")]
    [MaxFileSize(2_000_000)]
    [ImageWidth(400)]
    [ImageHeight(400)]
    [StringLength(1000, MinimumLength = 1)]
    public string Logo { get; set; }
}

Provider Setup

S3 (public or private)

Both S3PublicStorageService and S3PrivateStorageService share a single IAmazonS3 registration.

appsettings.json:

{
  "AppSettings": {
    "Spiderly.Shared": {
      "S3BucketName": "my-bucket",
      "S3PublicEndpoint": "https://cdn.example.com"
    }
  }
}

S3PublicEndpoint is the base URL S3PublicStorageService uses to format returned URLs as {S3PublicEndpoint}/{key}. The setting is unused by S3PrivateStorageService. S3 credentials (S3AccessKey, S3SecretKey, S3ServiceUrl) live in your application's settings, not in Spiderly.Shared.

DI registration (your AppServiceExtensions or equivalent):

services.AddSingleton<IAmazonS3>(sp =>
{
    IConfiguration configuration = sp.GetRequiredService<IConfiguration>();
    AmazonS3Config s3Config = new AmazonS3Config
    {
        ServiceURL = configuration.GetValue<string>($"{Spiderly.Shared.Settings.ConfigurationSection}:S3ServiceUrl"),
        ForcePathStyle = true,
        AuthenticationRegion = "auto",
    };

    return new AmazonS3Client(
        new BasicAWSCredentials(
            configuration.GetValue<string>($"{Spiderly.Shared.Settings.ConfigurationSection}:S3AccessKey"),
            configuration.GetValue<string>($"{Spiderly.Shared.Settings.ConfigurationSection}:S3SecretKey")
        ),
        s3Config
    );
});

services.AddTransient<S3PublicStorageService>();
services.AddTransient<S3PrivateStorageService>();

The source generator emits _deps.ServiceProvider.GetRequiredService<TConcrete>() per blob property, so each adapter you reference must be registered by its concrete type.

Disk (local development)

No appsettings entries needed. Files are stored under {CurrentDirectory}/FileStorage.

services.AddTransient<DiskStorageService>();

DiskStorageService is intended for development. In a Linux/Docker production deployment the host filesystem is ephemeral and not shared across replicas; switch to S3PublicStorage / S3PrivateStorage (or a custom adapter) for production.

Custom Adapters

Spiderly ships only the three adapters above. Other backends (Cloudinary, Azure Blob, Backblaze B2, on-prem MinIO, …) are written by the consumer.

Step 1 — Implement IFileManager

public class CloudinaryStorageService : IFileManager
{
    public Task<string> UploadFileAsync(...)             { /* impl */ }
    public Task DeleteNonActiveBlobs(...)                { /* impl */ }
    public Task<string> GetFileDataAsync(string key)     { /* impl */ }
    public Task<string> MoveBlobToEntityPathAsync(...)   { /* impl */ }
    public Task DeleteNonActiveEditorImages(...)         { /* impl */ }
}

Step 2 — Subclass StorageAttribute

Pass your service type to the base constructor:

public sealed class CloudinaryStorageAttribute : StorageAttribute
{
    public CloudinaryStorageAttribute() : base(typeof(CloudinaryStorageService)) { }
}

Step 3 — Register the service

services.AddTransient<CloudinaryStorageService>();

Step 4 — Use the attribute on entity properties

public class User : BusinessObject<long>
{
    [CloudinaryStorage]
    [AcceptedFileTypes("image/*")]
    [StringLength(500, MinimumLength = 1)]
    public string Photo { get; set; }
}

The source generator detects custom storage attributes by the convention "attribute simple name ends with Storage" and treats the property as a blob automatically. Field-name resolution in the generator is currently hard-coded for the three built-ins; for custom adapters in auto-generated CRUD, you may need to inject your custom service directly into hand-written upload paths until Spiderly's source generator gains symbol-level resolution of StorageAttribute.ServiceType.

Generated Upload Pipeline

Upload flow

1. Client sends POST /api/{Entity}/Upload{Property}For{Entity} with the file
2. OnBefore{Property}BlobFor{Entity}UploadIsAuthorized() hook runs
3. Authorization check (insert vs update based on entity ID)
4. File size validation — [MaxFileSize] if set, otherwise 20 MB default
5. MIME-type + content signature validation — [AcceptedFileTypes] is required on every blob property and must declare at least one MIME-typed value (e.g. [AcceptedFileTypes("image/jpeg", "image/png", "image/webp", "image/avif")]). If it is missing or contains only extension values, the source generator emits build error SPIDERLY014. MIME entries match the declared Content-Type exactly, or by prefix for type wildcards ("image/*"); extension entries (".pdf" — leading dot) only widen the UI file picker, so always pair them with a MIME type. The content is then verified against the declared type: binary types by magic-byte inspection, and image/svg+xml (a text format with no magic bytes) structurally — the document must parse as XML with an <svg> root, and active content (script elements, on* event attributes, javascript: hrefs, foreignObject) is rejected. Spoofing the header does not bypass validation. Failures throw BusinessException, so the user sees the specific localized message (FileTypeNotAllowed, FileContentDoesNotMatchType, FileContainsActiveContent, FileIsEmpty, FileSizeExceeded).
6. OnBefore{Property}BlobFor{Entity}IsUploaded() hook runs — for raster images, this validates dimensions and optimizes; SVG and non-image files pass through raw
7. File is uploaded to the storage adapter resolved per the property's [*Storage] attribute
8. The file identifier (key or URL) is returned to the client

Rate Limiting

All generated Upload*For* endpoints are decorated with [EnableRateLimiting(SpiderlyRateLimitPolicies.BlobUpload)]. Calling spiderly.AddRateLimiting() in your AddSpiderly(...) setup registers the policy with a default of 20 requests per minute per IP. Override the policy in your own Configure<RateLimiterOptions> call to tune the limit without forking Spiderly.

Default image processing

For raster image files (image/* except image/svg+xml — see Helper.IsOptimizableImage), the default OnBefore{Property}BlobFor{Entity}IsUploaded hook:

1. Validates dimensions — if [ImageWidth] or [ImageHeight] are set, checks exact pixel dimensions
2. Optimizes — converts to WebP format at 85% quality using SixLabors.ImageSharp

SVG is a vector text format ImageSharp cannot decode, so it skips both steps and uploads as-is; its safety is enforced by the structural validation in the upload flow above.

File Processing Hooks

All hooks are virtual methods on the generated entity service class (e.g., ProductServiceGenerated). Override them in your entity service class (e.g., ProductService) to customize behavior.

Example: custom image optimization

Override the optimization hook to resize images before storage:

public override async Task<byte[]> OptimizeImageForLogoOfBrand(
    Stream stream, IFormFile file, int id)
{
    return await Helper.OptimizeImage(
        stream,
        newImageSize: new Size(400, 400),
        quality: 90
    );
}

Example: skip optimization for a specific property

public override async Task<byte[]> OptimizeImageForBannerOfHomePage(
    Stream stream, IFormFile file, long id)
{
    return await Helper.ReadAllBytesAsync(stream);
}

Displaying Files

How uploaded files appear in DTOs depends on the storage adapter.

DTO generation

For every blob property, Spiderly generates a companion {Property}Data field on the DTO:

// Entity:
public string ProfilePicture { get; set; }

// Generated DTO:
public string ProfilePicture { get; set; }     // storage key or URL
public string ProfilePictureData { get; set; }  // file content for display

What {Property}Data contains

In the Angular admin panel, spiderly-file handles this automatically. It uses the [isUrlFileData] input (auto-generated) to determine how to render the preview.

Where {Property}Data is populated

Only single-row reads populate base64 blob data. Get{Entity}DTO — the read behind the details form — fills it; the list reads GetPaginated{Entity}List and Get{Entity}DTOList leave it null for [DiskStorage] and [S3PrivateStorage] properties.

That is deliberate: filling it in a list costs one storage round-trip and one base64 encode per row, on the request path of every table page, and no list surface renders file content. A table over a few hundred rows with a private blob column spends most of its response time downloading bytes the client throws away.

[S3PublicStorage] properties are unaffected — their {Property}Data is the URL the query already selected, so it is passed through everywhere, lists included.

Descriptive, SEO-friendly Keys

All adapters place uploaded blobs under a hierarchical, entity-scoped key:

{KeyPrefix}/{ObjectId}/{FileSegment}.{ext}

Both parts are customizable per property, and for [S3PublicStorage] they matter beyond aesthetics: the key is the public URL — the thing search engines index for your images and the thing every log and bucket listing shows you.

KeyPrefix — the path

KeyPrefix defaults to {EntityName}/{PropertyName} (editor-image properties: {EntityName}/{PropertyName}Image). Override it per property for short, lowercase, human-readable paths:

[S3PublicStorage(KeyPrefix = "products")]
[AcceptedFileTypes("image/*")]
[StringLength(1000, MinimumLength = 1)]
public string Url { get; set; }

The prefix is also the listing scope for save-time cleanup and staging promotion, so it must be unique per blob property, no prefix may be a path-parent of another, and custom values must be lowercase ASCII kebab-case segments — all enforced at build time by SPIDERLY030.

Descriptive file names — the GetBlobDescriptiveName hook

By default the file segment is a bare GUID. Override GetBlobDescriptiveNameFor{Property}Of{Entity}(id) on your entity service to give uploads a human/SEO-readable name — return the entity's slug, its display name, or a parent's (return empty for "no name"):

public class ProductMediaService : ProductMediaServiceGenerated
{
    public override async Task<string> GetBlobDescriptiveNameForUrlOfProductMedia(long id) =>
        await _deps.Context.DbSet<ProductMedia>()
            .Where(x => x.Id == id)
            .Select(x => x.Product.Slug)
            .SingleOrDefaultAsync() ?? "";
}

Whatever you return is slugified defensively — lowercased, diacritics transliterated (š→s, ü→u, plus the stroked letters and ligatures Unicode decomposition cannot fold: đ→dj, ł→l, ø→o, æ→ae, ß→ss), separator runs collapsed, and capped at 200 characters on a word boundary rather than mid-word. So a raw display name is as safe to return as a clean slug — and every one of those rules is configurable. The resulting key:

products/84512/cordless-drill-gsb-13-re-3f9a21c4.webp

The 8-character random suffix is load-bearing, not decoration: blobs are served with Cache-Control: immutable, so every upload must mint a fresh key or replaced content would be cached stale for up to a year. The hook is called eagerly on direct uploads (real entity id) and lazily at staged-blob promotion — the common no-staged-upload save never pays the lookup.

Naming policy — BlobKeyOptions

The defaults above encode judgements that are not universal, so each is a knob. Register it like any options class; anything you leave alone keeps its default.

services.Configure<BlobKeyOptions>(options =>
{
    options.MaxSlugLength = 80;
    options.Transliterations['đ'] = "d";   // Vietnamese: đ is the letter d, not the digraph dj
});

Honest extensions

The default optimize hook transcodes raster images to WebP, and a consumer override may produce any format — so before the key is built, the file name's extension is re-aligned with the bytes actually stored (by magic-byte detection, Helper.AlignExtensionWithContent). An admin uploading photo.jpg gets a .webp key served as image/webp; the key never lies about its content.

Two cases are deliberately left alone. SVG keeps its extension: it is a text format with no magic bytes, so detection can only see the generic XML underneath it, and renaming logo.svg to logo.xml would have it served as text/xml and stop rendering in an <img>. And a name that already agrees with any detected type is kept as-is, so a container format is never renamed to its generic parent (an .avif is also a generic ISO ftyp box; a .docx is also a .zip).

Storage Lifecycle and Orphan Cleanup

Insert flow — staging prefix

When a user uploads a file for an entity that doesn't exist yet (insert), the entity ID is 0. Spiderly routes these uploads to a temporary staging prefix:

{KeyPrefix}/_tmp/{UploadGuid}/{BlobGuid}.{ext}

Once the entity is saved and has a real ID, the generated save code calls IFileManager.MoveBlobToEntityPathAsync(...), which copies the blob to its permanent key ({KeyPrefix}/{realId}/{FileSegment}.{ext} — this is where the descriptive name is applied, since no trusted entity existed at upload time), deletes the staging source, and updates the DB column. The client never sees the staged path.

Update flow — replace and clean

When a user replaces a file on an existing entity:

1. User uploads a new file → new key/URL is returned
2. User saves the entity with the new key/URL
3. After SaveChangesAsync(), the generated code calls DeleteNonActiveBlobs() on the storage adapter
4. The adapter lists all files under {KeyPrefix}/{id}/ and deletes everything except the active file

This design is intentional — files are uploaded before the entity is saved (so the upload endpoint works independently). Cleanup only happens at save time, which means refreshing the page without saving won't lose the old file.

Opting out — [RetainReplacedBlobs]

Save-time cleanup assumes the entity is the only holder of the blob's URL. When copies of the URL outlive the entity's current value — order-line snapshots, sent emails, exports — deleting the replaced bytes breaks those historical references. Put [RetainReplacedBlobs] on the property to keep every replaced blob:

[S3PublicStorage(KeyPrefix = "products")]
[RetainReplacedBlobs] // order lines snapshot this URL at checkout
[AcceptedFileTypes("image/*")]
[StringLength(1000, MinimumLength = 1)]
public string Url { get; set; }

An orphaned blob costs fractions of a cent and is invisible; a deleted-but-referenced blob breaks a customer-facing record. When one side of the trade is cents, don't optimize the cents.