Skip to content

Implementing streamed tiles system for big georeferenced images #205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Implementing streamed tiles system for big georeferenced images #205

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6a74ab1
Checkpoint 1 is close
Fletterio Jul 23, 2025
e523868
Off by one error fix
Fletterio Jul 24, 2025
a54f6c6
Fix tile offsets for upload
Fletterio Jul 24, 2025
c3f7d04
Skeleton done but currently bugged, some byte offset is wrong (relate…
Fletterio Jul 28, 2025
7a5e948
Fix square bytes computation
Fletterio Jul 29, 2025
52d947d
Checkpoint 1!
Fletterio Jul 30, 2025
665559b
Save before merge
Fletterio Jul 31, 2025
a24281a
Merge + incorporate obb shrinking at the edges
Fletterio Aug 7, 2025
fc2d504
Bug: using uploaded uvs seems to stretch/not shrink along v direction
Fletterio Aug 8, 2025
e61e389
Fixed y-axis bug
Fletterio Aug 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
220 changes: 197 additions & 23 deletions 62_CAD/DrawResourcesFiller.cpp
View file
Open in desktop

Large diffs are not rendered by default.

43 changes: 38 additions & 5 deletions 62_CAD/DrawResourcesFiller.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,39 @@ struct DrawResourcesFiller
geometryInfo.getAlignedStorageSize();
}
};

// @brief Used to load tiles into VRAM, keep track of loaded tiles, determine how they get sampled etc.
struct StreamedImageManager
{
friend class DrawResourcesFiller;
constexpr static uint32_t TileSize = 128u;

StreamedImageManager(GeoreferencedImageParams&& _georeferencedImageParams);

struct TileUploadData
{
core::vector<StreamedImageCopy> tiles;
OrientedBoundingBox2D worldspaceOBB;
float32_t2 minUV;
float32_t2 maxUV;
};

TileUploadData generateTileUploadData(const float64_t3x3& NDCToWorld);

// This and the logic they're in will likely change later with Toroidal updating
protected:
GeoreferencedImageParams georeferencedImageParams;
uint32_t2 maxResidentTiles = {};
private:
ImageType imageType;
uint32_t2 minLoadedTileIndices = {};
uint32_t2 maxLoadedTileIndices = {};
uint32_t2 maxImageTileIndices = {};
float64_t2x3 world2Tile = {};
// Worldspace OBB that covers the top left `maxResidentTiles.x x maxResidentTiles.y` tiles of the image.
// We shift this OBB by appropriate tile offsets when loading tiles
OrientedBoundingBox2D fromTopLeftOBB = {};
};

DrawResourcesFiller();

Expand Down Expand Up @@ -342,7 +375,7 @@ struct DrawResourcesFiller
* @return true if the image was successfully cached and is ready for use; false if allocation failed.
* [TODO]: should be internal protected member function.
*/
bool ensureGeoreferencedImageAvailability_AllocateIfNeeded(image_id imageID, const GeoreferencedImageParams& params, SIntendedSubmitInfo& intendedNextSubmit);
bool ensureGeoreferencedImageAvailability_AllocateIfNeeded(StreamedImageManager& manager, SIntendedSubmitInfo& intendedNextSubmit);

// [TODO]: should be internal protected member function.
bool queueGeoreferencedImageCopy_Internal(image_id imageID, const StreamedImageCopy& imageCopy);
Expand All @@ -351,7 +384,7 @@ struct DrawResourcesFiller
void addImageObject(image_id imageID, const OrientedBoundingBox2D& obb, SIntendedSubmitInfo& intendedNextSubmit);

// This function must be called immediately after `addStaticImage` for the same imageID.
void addGeoreferencedImage(image_id imageID, const GeoreferencedImageParams& params, SIntendedSubmitInfo& intendedNextSubmit);
void addGeoreferencedImage(StreamedImageManager& manager, const float64_t3x3& NDCToWorld, SIntendedSubmitInfo& intendedNextSubmit);

/// @brief call this function before submitting to ensure all buffer and textures resourcesCollection requested via drawing calls are copied to GPU
/// records copy command into intendedNextSubmit's active command buffer and might possibly submits if fails allocation on staging upload memory.
Expand Down Expand Up @@ -596,7 +629,7 @@ struct DrawResourcesFiller
bool addImageObject_Internal(const ImageObjectInfo& imageObjectInfo, uint32_t mainObjIdx);;

/// Attempts to upload a georeferenced image info considering resource limitations (not accounting for the resource image added using ensureStaticImageAvailability function)
bool addGeoreferencedImageInfo_Internal(const GeoreferencedImageInfo& georeferencedImageInfo, uint32_t mainObjIdx);;
bool addGeoreferencedImageInfo_Internal(const GeoreferencedImageInfo& georeferencedImageInfo, uint32_t mainObjIdx);

uint32_t getImageIndexFromID(image_id imageID, const SIntendedSubmitInfo& intendedNextSubmit);

Expand Down Expand Up @@ -660,9 +693,9 @@ struct DrawResourcesFiller
*
* @param[out] outImageParams Structure to be filled with image creation parameters (format, size, etc.).
* @param[out] outImageType Indicates whether the image should be fully resident or streamed.
* @param[in] georeferencedImageParams Parameters describing the full image extents, viewport extents, and format.
* @param[in] manager Manager for the georeferenced image
*/
void determineGeoreferencedImageCreationParams(nbl::asset::IImage::SCreationParams& outImageParams, ImageType& outImageType, const GeoreferencedImageParams& georeferencedImageParams);
void determineGeoreferencedImageCreationParams(nbl::asset::IImage::SCreationParams& outImageParams, StreamedImageManager& manager);

/**
* @brief Used to implement both `drawHatch` and `drawFixedGeometryHatch` without exposing the transformation type parameter
Expand Down
5 changes: 4 additions & 1 deletion 62_CAD/Images.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ struct GeoreferencedImageParams
uint32_t2 imageExtents = {};
uint32_t2 viewportExtents = {};
asset::E_FORMAT format = {};
image_id imageID;
// For now it's going to be fully resident in memory, later on it's probably going to be a streamer class most likely.
core::smart_refctd_ptr<ICPUImage> geoReferencedImage;
// TODO: Need to add other stuff later.
};

Expand Down Expand Up @@ -205,7 +208,7 @@ class ImagesCache : public core::ResizableLRUCache<image_id, CachedImageRecord>
struct StreamedImageCopy
{
asset::E_FORMAT srcFormat;
core::smart_refctd_ptr<ICPUBuffer> srcBuffer; // Make it 'std::future' later?
ICPUBuffer* srcBuffer; // Make it 'std::future' later?
asset::IImage::SBufferCopy region;
};

Expand Down
49 changes: 38 additions & 11 deletions 62_CAD/main.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ enum class ExampleMode
CASE_9, // DTM
CASE_10, // testing fixed geometry and emulated fp64 corner cases
CASE_11, // grid DTM
CASE_12, // Georeferenced streamed images
CASE_COUNT
};

Expand All @@ -80,10 +81,11 @@ constexpr std::array<float, (uint32_t)ExampleMode::CASE_COUNT> cameraExtents =
600.0, // CASE_8
600.0, // CASE_9
10.0, // CASE_10
1000.0 // CASE_11
1000.0, // CASE_11
10.0 // CASE_12
};

constexpr ExampleMode mode = ExampleMode::CASE_8;
constexpr ExampleMode mode = ExampleMode::CASE_12;

class Camera2D
{
Expand Down Expand Up @@ -133,7 +135,7 @@ class Camera2D

if (ev.type == nbl::ui::SMouseEvent::EET_SCROLL)
{
m_bounds = m_bounds + float64_t2{ (double)ev.scrollEvent.verticalScroll * -0.1 * m_aspectRatio, (double)ev.scrollEvent.verticalScroll * -0.1};
m_bounds = m_bounds + float64_t2{ (double)ev.scrollEvent.verticalScroll * -0.0025 * m_aspectRatio, (double)ev.scrollEvent.verticalScroll * -0.0025};
m_bounds = float64_t2{ core::max(m_aspectRatio, m_bounds.x), core::max(1.0, m_bounds.y) };
}
}
Expand Down Expand Up @@ -1263,6 +1265,8 @@ class ComputerAidedDesign final : public nbl::examples::SimpleWindowedApplicatio

gridDTMHeightMap = loadImage("../../media/gridDTMHeightMap.exr");

bigTiledGrid = loadImage("../../media/tiled_grid.exr");

// set diagonals of cells to TOP_LEFT_TO_BOTTOM_RIGHT or BOTTOM_LEFT_TO_TOP_RIGHT randomly
{
// assumption is that format of the grid DTM height map is *_SRGB, I don't think we need any code to ensure that
Expand Down Expand Up @@ -1311,7 +1315,8 @@ class ComputerAidedDesign final : public nbl::examples::SimpleWindowedApplicatio

mouse.consumeEvents([&](const IMouseEventChannel::range_t& events) -> void
{
m_Camera.mouseProcess(events);
if (m_window->hasMouseFocus())
m_Camera.mouseProcess(events);
}
, m_logger.get());
keyboard.consumeEvents([&](const IKeyboardEventChannel::range_t& events) -> void
Expand Down Expand Up @@ -1492,7 +1497,7 @@ class ComputerAidedDesign final : public nbl::examples::SimpleWindowedApplicatio
projectionToNDC = m_Camera.constructViewProjection();

// TEST CAMERA ROTATION
#if 1
#if 0
// double rotation = 0.25 * PI<double>();
double rotation = abs(cos(m_timeElapsed * 0.0004)) * 0.25 * PI<double>() ;
float64_t2 rotationVec = float64_t2(cos(rotation), sin(rotation));
Expand Down Expand Up @@ -3127,12 +3132,6 @@ class ComputerAidedDesign final : public nbl::examples::SimpleWindowedApplicatio
//printf("\n");
}

GeoreferencedImageParams geoRefParams = {};
geoRefParams.format = asset::EF_R8G8B8A8_SRGB;
geoRefParams.imageExtents = uint32_t2 (2048, 2048);
geoRefParams.viewportExtents = (m_realFrameIx <= 5u) ? uint32_t2(1280, 720) : uint32_t2(3840, 2160); // to test trigerring resize/recreation
// drawResourcesFiller.ensureGeoreferencedImageAvailability_AllocateIfNeeded(6996, geoRefParams, intendedNextSubmit);

LineStyleInfo lineStyle =
{
.color = float32_t4(1.0f, 0.1f, 0.1f, 0.9f),
Expand Down Expand Up @@ -3686,6 +3685,33 @@ class ComputerAidedDesign final : public nbl::examples::SimpleWindowedApplicatio
}
#endif
}
else if (mode == ExampleMode::CASE_12)
{
GeoreferencedImageParams tiledGridParams;
auto& tiledGridCreationParams = bigTiledGrid->getCreationParameters();
// Position at topLeft viewport
auto inverseViewProj = nbl::hlsl::inverse(m_Camera.constructViewProjection());
const float64_t3 topLeftViewportH = float64_t3(-1.0, -1.0, 1.0);
const static auto startingTopLeft = nbl::hlsl::mul(inverseViewProj, topLeftViewportH);
tiledGridParams.worldspaceOBB.topLeft = startingTopLeft;
// Get screen pixel to match 2 viewport pixels (to test at mip border) by choosing appropriate dirU
const float64_t3 topRightViewportH = float64_t3(1.0, -1.0, 1.0);
const static auto startingViewportLengthVector = nbl::hlsl::mul(inverseViewProj, topRightViewportH - topLeftViewportH);
const static auto dirU = startingViewportLengthVector * float64_t(bigTiledGrid->getCreationParameters().extent.width) / float64_t(2 * m_window->getWidth());
tiledGridParams.worldspaceOBB.dirU = dirU;
tiledGridParams.worldspaceOBB.aspectRatio = 1.0;
tiledGridParams.imageExtents = { tiledGridCreationParams.extent.width, tiledGridCreationParams.extent.height};
tiledGridParams.viewportExtents = uint32_t2{ m_window->getWidth(), m_window->getHeight() };
tiledGridParams.format = tiledGridCreationParams.format;
tiledGridParams.imageID = 6996;
tiledGridParams.geoReferencedImage = bigTiledGrid;

DrawResourcesFiller::StreamedImageManager tiledGridManager(std::move(tiledGridParams));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to have a manager that knows everything about the image placement and info, useful for tile calculation.
Just keep in mind that we want to separate the idea of a "loader" and "tileCaclulator".
One actually does loads from disks or wherever, the other calculates which tiles and which mip levels are needed.
One has access to image data, One just knows the resolution and basic information
One doesn't have info about image placement in the worldspace and the view projection, one does.
One is loader, one is tile calculator or SteamedImageManager or whatever you want to call it


drawResourcesFiller.ensureGeoreferencedImageAvailability_AllocateIfNeeded(tiledGridManager, intendedNextSubmit);

drawResourcesFiller.addGeoreferencedImage(tiledGridManager, inverseViewProj, intendedNextSubmit);
}
}

double getScreenToWorldRatio(const float64_t3x3& viewProjectionMatrix, uint32_t2 windowSize)
Expand Down Expand Up @@ -3758,6 +3784,7 @@ class ComputerAidedDesign final : public nbl::examples::SimpleWindowedApplicatio

std::vector<smart_refctd_ptr<ICPUImage>> sampleImages;
smart_refctd_ptr<ICPUImage> gridDTMHeightMap;
smart_refctd_ptr<ICPUImage> bigTiledGrid;

static constexpr char FirstGeneratedCharacter = ' ';
static constexpr char LastGeneratedCharacter = '~';
Expand Down
Loading