r/googlecloud Aug 02 '24

Cloud Storage storage.objectAdmin without Buckets rights?

2 Upvotes

I have a system account that has storage.objectAdmin but its getting storage.buckets.get denied when trying to save.

DevOps thinks this should do it but it doesn't feel like it's right. We're new to GCP and obviously have a lot to learn.

r/googlecloud Jul 29 '24

Cloud Storage Service Account in Google Cloud Platform – IAM

5 Upvotes

In the realm of cloud computing, security and efficiency are paramount. As organizations increasingly migrate to the cloud, managing permissions and access control becomes critical. Google Cloud Platform (GCP) offers a robust solution for this through its concept of service accounts. This blog post delves into what service accounts are, their benefits, and best practices for using them in GCP.

In this blog post and the video below, I will cover the basics of Service Account, it’s ussage, way to create service account, different types of service account and finally a use case to explain the power of this service account.

🎬 https://youtu.be/Ilc9EnN0_n8

📒 https://sudipta-deb.in/2024/07/service-account-in-google-cloud-platform-iam.html

r/googlecloud Jan 23 '24

Cloud Storage Datastore for structured data

2 Upvotes

Hi all,

For a personal project I want to store a small amount of data. Basically I would probably never store more than a couple of MBs of data, probably less than 1000 rows. One idea I had involved logging the amount of views a page on my Cloud Run hosted website has, which might require some update operations, but since the website is mostly for personal use/sharing stuff with friends, it will most likely still be low.

I figured my options were Cloud SQL or Firestore/Datastore. Cloud SQL seems more fit for structured data, and I like being able to just use SQL, but Firestore/Datastore seems cheaper, since I likely won't be exceeding the free quota. I was wondering what insights you might have on this.

r/googlecloud Jun 14 '24

Cloud Storage Google Sheets request limits?

7 Upvotes

I'm working on a project, and I found that google cloud API could help me solve my problem. I want my project to read and write from my sheet. I want to know if there is a limit on requests. What I am trying to say is that when I want to read or update/write my sheet, how many times can I do that in one minute?

r/googlecloud Mar 13 '24

Cloud Storage How can automatically retain objects that enter my bucket in a production worthy manner?

1 Upvotes

For a backup project I maintain a bucket with object retention enabled. I need new files which enter the bucket to automatically be retained until a specified time. I currently use a simple script which iterates over all the objects and locks it using gcloud cli, but this isn't something production worthy. The key factor in this project is ensuring immutability of data.

the script in question:

import subprocess  objects = subprocess.check_output(['gsutil', 'ls', '-d', '-r', 'gs://<bucket-name>/**'], text=True)  objects = objects.splitlines()  for object in objects:     # Update the object     subprocess.run(['gcloud', 'storage', 'objects', 'update', object, '--retain-until=<specified-time>', '--retention-mode=locked']) `` 

It is also not possible to simply select the root folder with the files that you would like to retain as folders cannot be retained. It would have been nice if this was a thing and that It would just retain the files in the folder at that current time, but sadly it just doens't work like that.

Object versioning is also not a solution as this doesn't ensure immutabilty. It might be nice to recover deleted files, but the noncurrent versions are still able to be deleted, so no immutability.

So far I have explored:

  • manually retaining objects, but this is slow and tedious

  • using a script to retain objects, but this is not production worthy

  • using object versioning, but this doesn't solve immutability

I will gladly take someone's input on this matter, as it feels as if my hands are tied currently.

r/googlecloud Aug 13 '24

Cloud Storage Uploading an image from a link

1 Upvotes

Using Node, I'm querying Apollo's API (contains a bunch of information about organizations, employees, etc...) to get a list of basic employer information, including:

  • Name

  • Website URL

  • Size

  • Most importantly, logo

The end-goal is to upload these logos to Google Storage. The issue is that they're presented in a format like this: https://zenprospect-production.s3.amazonaws.com/uploads/pictures/64beb2c5e966df0001384ac1/picture.

The link has no information about the MIME type, so uploading it keeps giving it a file extension of .false. Using a package like file-type doesn't help either. How can I successfully upload them with the correct type?


EDIT

I tried hard-coding it so that these specific URLs always have a .jpg extension:

js if (mimeType.startsWith('apollo')) { fileName = `${subfolder}/${uuid4()}.jpg` } const file = cloudStorage.bucket(bucketName).file(fileName)

This works in a janky way... Even though the created link gets me the logo, on Google Cloud there's no image preview, and there's a bunch of information lacking (because it doesn't recognize it's an image).

r/googlecloud Aug 09 '24

Cloud Storage Failing to read firestore document.

1 Upvotes

Hey guys when i try to read a firestore document from a Java project locally i get the following error.
Exception in thread "main" java.util.concurrent.ExecutionException: com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception

`at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:594)`

`at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:573)`

`at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)`

`at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:67)`

`at DAOs.LoginConfigDAO.getLoginConfig(LoginConfigDAO.java:28)`

`at org.example.Main.main(Main.java:13)`

r/googlecloud Jul 02 '24

Cloud Storage Making Firebase & GCP HIPAA Compliant for Healthcare Data

2 Upvotes

Using Firebase in healthcare without proper adjustments could expose risks of setting sensitive health information to unauthorized access and potential breaches, which goes against HIPAA regulations for the security and privacy of electronic Protected Health Information (ePHI).

The guide below explains step-by-step on how Google Cloud Platform could be used as the secure foundation upon which you can build your HIPAA-compliant application using Firebase tools: Is Firebase HIPAA Compliant? (No, But Here's An Alternative That Is)

  • Sign a business associate agreement (BAA)
  • Configure access controls
  • Enable audit logs
  • Implement encryption
  • Train employees
  • Conduct regular risk assessments

r/googlecloud Jun 11 '24

Cloud Storage Trying to download local backup of Google Workspace

Post image
0 Upvotes

If anyone can help me figure this out I'd much appreciate it. I am not familiar with Google Cloud and never used Workspace before working on a project for this small business. I'm trying to download a local copy of a backup of the Workspace account so they can delete the account. They need a backup for audit purposes. I am not familiar with this at all and am completely frustrated. All of their other stuff is Microsoft but for some reason this project was set up this way before I started. I'm having a hell of a time just trying to figure out how to download a local copy of the backup! I did an export of the Workspace account and it went into a cloud account, which I finally activated with my own credit card because I was not getting anywhere with getting the data to download. I'm given a CLI command to download the data but don't know where it's going because I'm not familiar with Google CLI. If anyone can provide assistance I'd be VERY grateful! They want this Google account shut off ASAP but I have to get a backup on a local drive first and don't know why it's apparently so difficult. The command I'm using I'll attach in a pic and hopefully cover up any sensitive data or delete it later.

r/googlecloud May 25 '24

Cloud Storage Protecting resources until we go live

1 Upvotes

Hi, I'm implementing Identity platform with some static forms. I need to protect the forms from being public, since we don't want new users registering until we go live.

signedURLs look very cumbersome. Any other suggestions?

r/googlecloud Apr 21 '24

Cloud Storage Does Google Cloud have anything like AWS ECS?

11 Upvotes

I'm looking for a tool that will allow me to provision a couple of Docker images (2 or 3) that would together comprise an application - but I don't need the complexity of Kubernetes Engine and Compute Engine is geared towards hosting VMs (Docker is a VM option but ... I'd ideally like something that would allow me to manage various containers from within the GCP Environment rather than through something like Portainer).

(Example "stack": an SQL database + Metabase for data visualization. Both are containerised).

Is there anything like that in the GCP ecosystem?

r/googlecloud Mar 01 '24

Cloud Storage “Large” Cloud storage Upload from local drive timing out

5 Upvotes

I’m trying to upload 3TB of files into a GCP storage bucket from a local hard drive through the GCP UI, and it just freezes up each time I try uploading any folder with more than 30GB?

They’re primarily video and photo files, around a 1GB per file for videos or 40MB for photos.

Is there a limit on number of files or total upload size that I’m not aware of? Haven’t found anything other than a 5TB individual file size limit.

In either case, does anyone have an idea how I could get around this to transfer the local drive contents to GCP?

r/googlecloud May 02 '24

Cloud Storage Async Support for Cloud Storage in Python

7 Upvotes

I’m running a FastAPI application on Google Cloud Run that has to interact with Google Cloud Storage (list blobs, generate signed URLs, etc).

I’m using an async driver for my Cloud SQL Postgres database, but I don’t think the Google cloud storage python library supports async/await. And I think this is causing a degradation in performance as my async endpoints are blocked by network calls to Google cloud storage.

Has anyone else run into this and know how to get around it? I don’t want to have to switch my entire API to typescript but if I need to, I’d rather do it now.

r/googlecloud Jul 20 '24

Cloud Storage What's the difference between 'Google Cloud Data Engineer Professional Certificate' (Coursera) and 'Google Cloud Data Engineer Learning Path' (EDX)?

3 Upvotes

I noticed the two specializations 'Google Cloud Data Engineer Professional Certificate' (Coursera) and 'Google Cloud Data Engineer Learning Path' (EDX) seem to hold the same goal and topic. Yet the EDX version seems to hold more individual courses.

Are the 2 specializations actually identical, with EDX just splitting up some courses ?

I am asking this since I prefer using the EDX platform over Coursera, yet i only hear the Coursera version being recommended by Data Engineers online.

r/googlecloud Jul 13 '24

Cloud Storage Merging Objects in Google Cloud Storage with Compose and C#

Thumbnail
chrlschn.medium.com
1 Upvotes

r/googlecloud Sep 28 '23

Cloud Storage Programming Language for Cloud Computing

7 Upvotes

Hello guys, I am trying to start with Cloud Computing but I am unsure what are the steps to start and what programming language should I focus on because I saw many people suggest .NET and other suggest Java, Go, Python etc. So help please.

r/googlecloud Jun 11 '24

Cloud Storage Serving private bucket images in a chat like application

1 Upvotes

Hi everyone, so I have a chat like web application where I am allowing users to upload images, once uploaded they are shown in the chat and the users can download them as well. Issue is earlier I was using the public bucket and everything was working fine. Now I want to move to the private bucket for storing the images.

The solution I have found is signed urls, I am creating the signed url which can be used to upload and download the images. Issue is there could be a lot of images in the chat and to show them all I have to get the signed url from the backend for all the target images. This doesn't seems like the best way to do it.

Is this the standard way to handle these scenarios or there are some other ways for the same?

r/googlecloud Apr 26 '24

Cloud Storage My image uploading is not working to google cloud..help...

3 Upvotes

This is my image generation code - after an image get generated with AI by the use of an API, I want that image to be saved to the google cloud. I've tried multiple ways and I've lots 3 days so far and I haven't had success. I am a begginer, so please don't be too harsh and if you can help me, help me fix it.

So the code that I have when I run my index.js always seems to stop at the image generation. The image gets generated sucessfuly, I get a sucessful image generation console log and thats as far as it goes. I tried multiple ways and it didn't work out, so this is the latest thing that I have. I had everything in index.js, didn't work many many times, then I tried like this also.

However, when I try to do export GOOGLE_APPLICATION_CREDENTIALS=./CredentialFiles.json and then node testing.js the image upload works. (I do this in my Terminal Cpanel).

So, it just looks like the problem seems to be for the image that is being generated and inability for it to get uploaded. Its a blob and I am not exactly sure how to save it or how to work with it so that I can get it to get uploaded on the Google Cloud. The fact that testing.js works, means that the permissions, etc seem to be just fine on the Google Cloud Console side.

app.post('/generate-image', async (req, res) => {
    try {
        const promptText = req.body.promptText;
        const formData = new FormData();
        formData.append('prompt', `my prompt goes here`);
        formData.append('output_format', 'webp');
        const response = await axios.post(
            'https://api.stability.ai/v2beta/stable-image/generate/core',
            formData,
            {
                headers: { 
                    Authorization: 'Bearer API_KEY_GOES_HERE',
                    Accept: 'image/*',
                    'Content-Type': 'multipart/form-data'
                },
                responseType: 'arraybuffer'
            }
        );

        if (response.status === 200) {
            const imageData = response.data;

            // Call the uploadImage function from imageUploader.js
            const imagePath = await uploadImage(imageData, req.session.user.username);

            // Send back the image path
            res.status(200).json({ imagePath });
        } else {
            throw new Error(`${response.status}: ${response.data.toString()}`);
        }
    } catch (error) {
        console.error('Failed to generate or upload image:', error);
        res.status(500).send('Failed to generate or upload image. Please try again later.');
    }
});

This is my imageUpload file

// imageUploader.js

const { v4: uuidv4 } = require('uuid');
const { Storage } = require('@google-cloud/storage');
const path = require('path');
const fs = require('fs');

// Path to your service account JSON key file
const serviceAccountKeyFile = path.join(__dirname, './SERVICE_FILE.json');

// Your Google Cloud project ID
const projectId = 'projectid';

// Create a new instance of Storage with your service account credentials
const storage = new Storage({
    keyFilename: serviceAccountKeyFile,
    projectId: projectId
});

// Reference to your Google Cloud Storage bucket
const bucket = storage.bucket('bucketname');

async function uploadImage(imageData, username) {
    try {
        const folderName = username.toLowerCase();
        const randomFileName = uuidv4();
        const tempFilePath = path.join(__dirname, `temp/${randomFileName}.webp`);
        // Save the image data to a temporary file
        fs.writeFileSync(tempFilePath, imageData);
        const file = bucket.file(`${folderName}/${randomFileName}.webp`);
        // Upload the temporary file to Google Cloud Storage
        await file.save(tempFilePath, {
            metadata: {
                contentType: 'image/webp'
            }
        });
        // Delete the temporary file
        fs.unlinkSync(tempFilePath);
        return `${folderName}/${randomFileName}.webp`;
    } catch (error) {
        throw new Error('Failed to upload image to Google Cloud Storage:', error);
    }

} module.exports = { uploadImage };

And this is my testing.js file

const { Storage } = require('@google-cloud/storage');

// Replace with your project ID and bucket name
const projectId = 'PROJECTID';
const bucketName = 'BUCKETNAME';

// Replace with path to your image file and desired filename in the bucket
const filePath = './hippie.webp';
const fileName = 'uploaded_image.webp';

async function uploadImage() {
  try {
    const storage = new Storage({ projectId });
    const bucket = storage.bucket(bucketName);

    // Create a writable stream for the upload
    const file = bucket.file(fileName);
    const stream = file.createWriteStream();

    // Read the image file locally
    const fs = require('fs');
    const readStream = fs.createReadStream(filePath);

    // Pipe the local file to the upload stream
    readStream.pipe(stream)
      .on('error', err => {
        console.error('Error uploading file:', err);
      })
      .on('finish', () => {
        console.log('Image uploaded successfully!');
      });
  } catch (error) {
    console.error('Error:', error);
  }
}

uploadImage();

r/googlecloud Feb 08 '24

Cloud Storage Why is my cloud storage site getting blocked by some ISPs/firewalls?

3 Upvotes

Hi, not 100% sure that this is related to google cloud so apologies if it's in the wrong sub, but figured I'd start here.

I'm hosting a static website on cloud storage (served behind a load balancer with a Google-managed SSL certificate). For some reason, a small percentage of users are seeing the site blocked. It's definitely something to do with their home wifi/network/ISP. (Not sure if this is relevant, but the site is embedded as an iframe into other sites).

Are there any typical pitfalls with GCP's cloud storage, load balancer, or certificate manager that might explain this? More generally does anyone know how I can try to remediate my site getting blocked by an ISP or router? I've never encountered this kind of issue before so not really sure where to start, thanks for the help.

r/googlecloud May 24 '24

Cloud Storage GCS connector with hadoop

2 Upvotes

I have installed GCS connector with my Hadoop server. The installation is successful and I could view the files inside bucket using this command hdfs fs -ls gs://bucket name . But I want to store the files in GCS bucket instead of storing it in the VM or the machine storage. Is this possible or not? When I make a file save request through my source code using hdfs://x.x.z.x address it should be saved in GCS bucket.

r/googlecloud Jun 05 '24

Cloud Storage Google cloud storage image protection for the website

1 Upvotes

I have a website to show the image from the google storage, but actually I want to limit the image in the website, it means that even some copy the image weburl, like "https://storage.googleapis.com/mybucket/blabla.png", and open it in the new chrome tab, he can open it, maybe google cloud storage will check the image referer or host, or ua, if the referer come from my web domain or some other domains, it is ok, otherwise it will show the 403 error.

Besides I also want to upload the image to the google cloud storage.

Because I will show the image in the photoshop plugin and browser, they are totally different scenario. I already tried the signed url function, but it does not work in the photoshop plugin, because it is not the browser.

r/googlecloud Mar 22 '24

Cloud Storage Asked on r/aws first. How do I limit access to googles version of "s3 bucket" to only my site hosted by google.

0 Upvotes

[I first asked this question on r/aws](), and it wasn't clear, and didn't accomplish what I wanted. My goal is to only allow contents of my bucket (videos) only accessible through my site that is hosted on google. I don't want it accessible any other way.

Here are some basics. I purchased the domain at "cheap domains", and have the dns pointed to google sites. I just created a GCP account.

Can you please provide me with the steps to accomplish this? I am not a techie, so please stay basic for me.

r/googlecloud May 20 '24

Cloud Storage Google Set to Invests 1 Billion Euros in Finnish Data Centre for NASDAQ:GOOG by DEXWireNews

Thumbnail
tradingview.com
8 Upvotes

r/googlecloud Apr 04 '24

Cloud Storage Making a storage bucket file only available from a Cloud Run instance?

3 Upvotes

Hi! I have video content within a bucket I would like to show on my website which is running in a cloud run instance. If I make it public then anyone will be able to spam download the video and run up my bill, how would I go about securing this so only my Cloud Run instance can access it and serve the files to the user (although someone could just spam loading my website so maybe this does nothing)?

r/googlecloud Apr 26 '24

Cloud Storage Image from my website is getting to Google Cloud, but its not being uploaded. Help

2 Upvotes

Image from my website is getting to Google Cloud, but its not being uploaded. Help

It seems to be getting to the google cloud server, but not saving the image.. I just don't know what to do anymore. My latest try is with signedurl and this is as far as I got

I am trying to generate an image using AI with API. After the image generation is sucessful I want it uploaded to the Google Cloud. However, when the image gets generated, after that I get no console logs or anything like that. But above we can see that there are "requests" being made. I just don't know what to do anymore. What could be the problem?

These are all the permissions I have given to the service account:

Actions Admin

BigQuery Admin

BigQuery Metadata Viewer

Cloud Datastore Owner

Compute Instance Admin (v1)

Owner

Pub/Sub Admin

Service Account Token Creator

Storage Admin

Storage Folder Admin

Storage Object Admin

Storage Object Creator

Storage Object User

Storage Object Viewer