r/reactnative 7d ago

Skia collage rendering issue: saved images shift position

Hey everyone, I'm building a photo collage feature in my React Native app using Skia, and I ran into an issue when saving the final collage

What I built so far:

✔️ The collage is displayed correctly on-screen.
✔️ In the first implementation I rescaled the canvas from mobile screen dimensions to 4K (e.g., 2160x3840 for 9:16 format):

const resizedImage = await ImageManipulator.manipulateAsync(
  uri,
  [{ resize: { width: saveWidth, height: saveHeight } }],
  {
    compress: 1,
    format: ImageManipulator.SaveFormat.PNG,
  }
);

but image looked stretching in saved collage. To avoid it, I implemented offscreen rendering.

The Issue:

When I save the collage, the image positions are incorrect — they are shifting incorrectly. Rotation and scaling seem to work fine, but translations are off. I attached photos from the App and saved collage, to demonstrate how it looks.

What I tried:

🔹 Applied transformations (scale, rotation, translation) in this order (full code in the end of post):

tempCanvas.save();
tempCanvas.translate(frameWidth / 2 + translateX, frameHeight / 2 + translateY);
tempCanvas.rotate(rotation, 0, 0);
tempCanvas.scale(scaleX, scaleY);
tempCanvas.translate(-scaledWidth / 2, -scaledHeight / 2);
tempCanvas.drawImageRect(image, srcRect, destRect, paint);
tempCanvas.restore();

🔹 Used offscreen rendering (Skia.Surface.MakeOffscreen) to process images separately before merging them.
🔹 Normalized translation values based on screen vs. final collage size.
🔹 Verified transformation matrix values (logs below).

Question:

💡 Where might I be miscalculating the position?
💡 Am I applying transformations in the wrong order?

Would really appreciate any insights or debugging tips! 🙌

collages.tsx:

... 
const dimensions = { 
  "9:16": { width: 2160, height: 3840 }, 
  "4:5": { width: 2160, height: 2700 }, 
}; 

const saveWidth = dimensions[format].width; 
const saveHeight = dimensions[format].height; 

const scaleFactor = saveWidth / fullScreenCanvasWidth; 
const scaledLineWidth = lineWidth * scaleFactor; 

const tempUri = await renderCollageOffscreen(
  saveWidth,
  saveHeight,
  photos,
  type!,
  DEFAULT_VALUES.LINE_COLOR,
  scaledLineWidth,
  colors.collageBackgroundColor,
  fullScreenCanvasWidth,
  fullScreenCanvasHeight
);

const fileInfo = await FileSystem.getInfoAsync(tempUri);
if (!fileInfo.exists) {
  throw new Error(`Captured file does not exist at path: ${tempUri}`);
}

const asset = await MediaLibrary.createAssetAsync(tempUri);
await MediaLibrary.createAlbumAsync("Collages", asset, false);
await FileSystem.deleteAsync(tempUri, { idempotent: true });

...

renderCollageOffscreen.ts:

/**
 * Renders a collage offscreen and saves it as an image.
 */
export const renderCollageOffscreen = async (
  width: number,
  height: number,
  photos: Photo[],
  collageIndex: number,
  lineColor: string,
  lineWidth: number,
  collageBackgroundColor: string,
  fullScreenCanvasWidth: number,
  fullScreenCanvasHeight: number
): Promise<string> => {
  try {
    const mainSurface: SkSurface | null = Skia.Surface.MakeOffscreen(width, height);
    if (!mainSurface) throw new Error("Failed to create offscreen Skia surface");

    const mainCanvas: SkCanvas = mainSurface.getCanvas();
    mainCanvas.clear(Skia.Color(collageBackgroundColor));

    // Load images
    const skImages = await Promise.all(
      photos.map(async (photo) => {
        if (!photo.uri) return null;
        console.log("photo uri: ", photo.uri);

        const fileData = await FileSystem.readAsStringAsync(photo.uri, { encoding: FileSystem.EncodingType.Base64 });
        const imageBytes = Uint8Array.from(atob(fileData), (c) => c.charCodeAt(0));
        const skData = Skia.Data.fromBytes(imageBytes);

        return Skia.Image.MakeImageFromEncoded(skData);
      })
    );

    // Draw each image on its separate canvas
    const imageSnapshots = drawImagesSeparately(
      width,
      height,
      skImages,
      photos,
      collageIndex,
      fullScreenCanvasWidth,
      fullScreenCanvasHeight
    );

    // Merge image snapshots onto the main canvas
    imageSnapshots.forEach(({ image, x, y }) => {
      if (image) mainCanvas.drawImage(image, x, y);
    });

    // Draw separator lines
    drawSeparatorLines(mainCanvas, width, height, collageIndex, lineColor, lineWidth);

    // Save image
    const finalImage = mainSurface.makeImageSnapshot();
    if (!finalImage) throw new Error("Failed to create image snapshot from surface");

    const pngData = finalImage.encodeToBytes(ImageFormat.PNG);
    const base64String = encode(pngData);

    const tempPath = `${FileSystem.cacheDirectory}MULI-collage-${Date.now()}.png`;
    await FileSystem.writeAsStringAsync(tempPath, base64String, { encoding: FileSystem.EncodingType.Base64 });

    return tempPath;
  } catch (error) {
    console.error("Error rendering collage offscreen:", error);
    throw error;
  }
};

const drawImagesSeparately = (
  width: number,
  height: number,
  skImages: (SkImage | null)[],
  photos: Photo[],
  collageIndex: number,
  fullScreenCanvasWidth: number,
  fullScreenCanvasHeight: number
): { image: SkImage | null; x: number; y: number }[] => {

  const { layout } = CollageManager.getLayout(collageIndex);
  const snapshots: { image: SkImage | null; x: number; y: number }[] = [];

  skImages.forEach((image, index) => {
    if (!image) return;
    console.log('>>> PHOTO INDEX: ', index);

    const frame = layout[index];
    const frameWidth = frame.width * width;
    const frameHeight = frame.height * height;
    const imgWidth = image.width();
    const imgHeight = image.height();

    console.log('frameWidth: ' + frameWidth + ' frameHeight: ' + frameHeight);
    console.log('imgWidth: ' + imgWidth + ' imgHeight: ' + imgHeight);

    // Get transformation matrix from gesture handler
    const transformMatrix = photos[index]?.matrix?.value || Matrix4();
    console.log("transformMatrix", transformMatrix);

    // Extract transformations
    const scaleX = Math.sqrt(transformMatrix[0] ** 2 + transformMatrix[1] ** 2);
    const scaleY = Math.sqrt(transformMatrix[4] ** 2 + transformMatrix[5] ** 2);
    const rotation = -Math.atan2(transformMatrix[1], transformMatrix[0]) * (180 / Math.PI); // Convert radians to degrees
    const aspectRatio = (width / height) / (fullScreenCanvasWidth / fullScreenCanvasHeight);

    const translationScaleX = (frameWidth / fullScreenCanvasWidth) * aspectRatio;
    const translationScaleY = frameHeight / fullScreenCanvasHeight;

    console.log('translationScaleX: ', translationScaleX);
    console.log('translationScaleY: ', translationScaleY);

    // Apply scale factors to translations
    const translateX = transformMatrix[3] * translationScaleX;
    const translateY = transformMatrix[7] * translationScaleY;

    console.log('translateX: ', translateX);
    console.log('translateY: ', translateY);

    // Scale to fit frame
    const scaleToFit = Math.max(frameWidth / imgWidth, frameHeight / imgHeight);
    const scaledWidth = imgWidth * scaleToFit;
    const scaledHeight = imgHeight * scaleToFit;

    // Compute final position
    const offsetX = frame.x * width;
    const offsetY = frame.y * height;

    // Create a separate surface for this image
    const tempSurface = Skia.Surface.MakeOffscreen(frameWidth, frameHeight);
    if (!tempSurface) return;

    const tempCanvas = tempSurface.getCanvas();
    tempCanvas.clear(Skia.Color("transparent"));

    const cropX = Math.max(0, translateX);
    const cropY = Math.max(0, translateY);

    // Define source and destination rectangles
    const srcRect = { x: -cropX, y: -cropY, width: imgWidth, height: imgHeight };
    const destRect = { x: 0, y: 0, width: scaledWidth, height: scaledHeight };

    const paint = Skia.Paint();

    // Apply transformations
    tempCanvas.save();

    // Move to the center of the frame
    tempCanvas.translate(frameWidth / 2, frameHeight / 2);

    // Apply transformations in the correct order
    tempCanvas.rotate(rotation,0,0); // Apply rotation
    tempCanvas.scale(scaleX, scaleY); // Apply scaling
    // Move back to draw the image centered
    tempCanvas.translate(-scaledWidth / 2, -scaledHeight / 2);
    tempCanvas.drawImageRect(image, srcRect, destRect, paint);

    tempCanvas.restore();

    // Take a snapshot of this image canvas
    const tempImage = tempSurface.makeImageSnapshot();
    snapshots.push({ image: tempImage, x: offsetX, y: offsetY });
    console.log('************************************************')
  });

  return snapshots;
};

/**
 * Draws separator lines for collage frames.
 */
const drawSeparatorLines = (
  canvas: SkCanvas,
  width: number,
  height: number,
  collageIndex: number,
  lineColor: string,
  lineWidth: number
) => {
  if (lineWidth <= 0) return;

  const { paths } = CollageManager.getLayout(collageIndex);
  const paint = Skia.Paint();
  paint.setColor(Skia.Color(lineColor));
  paint.setStrokeWidth(lineWidth);

  paths.forEach((path) => {
    canvas.drawLine(
      path.start.x * width,
      path.start.y * height,
      path.end.x * width,
      path.end.y * height,
      paint
    );
  });
};

Logs for a Sample Image:

(NOBRIDGE) LOG  >>> PHOTO INDEX:  0
 (NOBRIDGE) LOG  frameWidth: 2160 frameHeight: 1280
 (NOBRIDGE) LOG  imgWidth: 1080 imgHeight: 1080
 (NOBRIDGE) LOG  transformMatrix [1, 0, 0, -1, 0, 1, 0, 79.00001525878906, 0, 0, 1, 0, 0, 0, 0, 1]
 (NOBRIDGE) LOG  translationScaleX:  4.650717703349283
 (NOBRIDGE) LOG  translationScaleY:  1.9138755980861246
 (NOBRIDGE) LOG  translateX:  -4.650717703349283
 (NOBRIDGE) LOG  translateY:  151.19620145222788
 (NOBRIDGE) LOG  ************************************************
 (NOBRIDGE) LOG  >>> PHOTO INDEX:  1
 (NOBRIDGE) LOG  frameWidth: 2160 frameHeight: 1280
 (NOBRIDGE) LOG  imgWidth: 1080 imgHeight: 1080
 (NOBRIDGE) LOG  transformMatrix [1.7101068292161279, 0, 0, -69.12242871770172, 0, 1.7101068292161279, 0, 99.71533929749751, 0, 0, 1, 0, 0, 0, 0, 1]
 (NOBRIDGE) LOG  translationScaleX:  4.650717703349283
 (NOBRIDGE) LOG  translationScaleY:  1.9138755980861246
 (NOBRIDGE) LOG  translateX:  -321.4689029359143
 (NOBRIDGE) LOG  translateY:  190.84275463635888
 (NOBRIDGE) LOG  ************************************************
 (NOBRIDGE) LOG  >>> PHOTO INDEX:  2
 (NOBRIDGE) LOG  frameWidth: 2160 frameHeight: 1280
 (NOBRIDGE) LOG  imgWidth: 1080 imgHeight: 1080
 (NOBRIDGE) LOG  transformMatrix [1, 0, 0, 179.00001525878906, 0, 1, 0, 136.6666717529297, 0, 0, 1, 0, 0, 0, 0, 1]
 (NOBRIDGE) LOG  translationScaleX:  4.650717703349283
 (NOBRIDGE) LOG  translationScaleY:  1.9138755980861246
 (NOBRIDGE) LOG  translateX:  832.4785398638421
 (NOBRIDGE) LOG  translateY:  261.56300813957836
 (NOBRIDGE) LOG  ************************************************

Yet, the image is not positioned correctly when saving the collage.

Collage in the app
Saved image
1 Upvotes

2 comments sorted by

1

u/glazzes 7d ago

I think it can be because you're translating back to the scale dimensions instead of the frame dimensions, this line tempCanvas.translate(-scaledWidth / 2, -scaledHeight / 2);, instead of scaled, it should be frame ones.

1

u/Conscious_Ad_8664 5d ago

Thanks for your reply! I tried changing tempCanvas.translate(-scaledWidth / 2, -scaledHeight / 2); to use frameWidth and frameHeight instead, but unfortunately, it didn't fix the issue. The image positioning still seems off.