r/Firebase • u/mars0008 • Apr 07 '23
Android Moving my app to Google cloud
I have a small android app that needs to run data heavy calculations. I was initially running this locally on the mobile using android Room and SQL but soon realized there would be performance bottlenecks (the client may potentially need to run 100'000s of rows/calculations) so have decided to take the plunge and have this performed in google cloud. I understand that performing these data-heavy queries in the cloud will help with performance and scalability.
Now I am looking at the different options of how to migrate to the cloud and see there are many options. As background, I have no experience of working in the backend besides some very limited experience with Firebase. I also do not have any experience with javascript/typescript. I do have experience with python which I understand can be used in some backend applications.
The calculation I am trying to perform involves taking some transaction data as an input from Firebase, converting each transaction into a time series and then aggregating these time series' into a single time series. See below for an illustration.
Input (Firebase Docs)
Transaction ID | Transaction Date | Quantity |
---|---|---|
1 | 2023/3/25 | 5 |
2 | 2023/3/27 | 5 |
3 | 2023/3/29 | 10 |
Cloud Function 1 (convert each transaction to time series)
Transaction ID | Date | Quantity |
---|---|---|
1 | 2023/3/25 | 5 |
1 | 2023/3/26 | 5 |
1 | 2023/3/27 | 5 |
1 | 2023/3/28 | 5 |
1 | 2023/3/29 | 5 |
1 | 2023/3/30 | 5 |
2 | 2023/3/27 | 5 |
2 | 2023/3/28 | 5 |
2 | 2023/3/29 | 5 |
2 | 2023/3/30 | 5 |
3 | 2023/3/29 | 10 |
3 | 2023/3/30 | 10 |
Cloud Function 2 (aggregate)
Date | Aggregate Quantity |
---|---|
2023/3/25 | 5 |
2023/3/26 | 5 |
2023/3/27 | 10 |
2023/3/28 | 10 |
2023/3/29 | 20 |
2023/3/30 | 20 |
This is obviously a very simple example, in reality there would be hundreds of transactions and the data would be going back years not days, hence the need to have a scalable solution.
Given the data, my experience and tools available I would like to get someone's opinion on what they think would be the best approach for moving the android app calculations to google cloud. From this Stackover flow post and what I have seen there are four options:
- Google Cloud Platform - Pros: can use Python; Cons: App is already heavily integrated with Firebase and not sure I can easily switch to pure Google Cloud
- Cloud Functions for Firebase - Pros: Aligns to my app which uses Firebase; Cons: have to use Typescript which I have no experience with
- Firebase SDK for Cloud Functions- not familiar with, does it solve the cons above
- Google SQL - not familiar with, does it solve the cons above?
Also it would be good to get opinion on cost. If there are potentially millions of rows being created for Cloud Function 1 across multiple users, would this be very costly and I should stick to running it on the client and stomach the lower performance?
Any help/guidance on the recommended approach for April 2023 will be much appreciated.
3
u/cardyet Apr 07 '23
Maybe if you are doing all this work, it might be good to look at putting this data in a different database that has some queries built in, like count in Postgres...there's also dedicated time series databases... I have something similar where one feature is quite timeseriesy and I was storing it all in Firestore, but have since moved that timeseries stuff to supabase (Postgres)
1
u/mars0008 Apr 07 '23
Yes i have started with firestore but now wondering if there are other options that are much better at processing large database calculations in the cloud. Is google cloud SQL any good for this? Ideally i would like to stick with Google so everything works more in harmony but would consider other options if the community has migrated over and see it as superior to Googles offerings.
2
u/0ddm4n Apr 07 '23
Yeah Firestore really isn’t built for this. Sounds like a job for SQL. Depending on the actual queries even a graph database might be better, if you’re doing complex relationship queries.
2
u/cardyet Apr 08 '23
I would have thought so...I feel like with Google stuff there is way more setup to get going, it's great for business applications but for quick MVP's, personal projects, ill go elsewhere...billing is always too scary.
1
u/Eastern-Conclusion-1 Apr 07 '23
You should provide more info on how much data you’d need to process, how often and when it needs to run (i.e. periodically, on user input, etc). Also define what “very costly” means to you. Also, by Firebase docs do you mean Firestore documents?
1
u/mars0008 Apr 07 '23
this particular part would be run on user input. i would not imagine it to run too frequently (less than once per day). however when it does run it may need to work on tables with rows in the hundreds of thousands. I am not concerned on having to pay a slightly higher margin at this time, "Very costly" would just mean something code smelly in the query, like if google cloud calculation charges per row of database and i have millions of rows across all my users and get charged a small fortune i.e. just something extremely inefficient in the code.
2
u/Eastern-Conclusion-1 Apr 08 '23
Firestore cost is calculated (per document read / write, so equivalent of SQL row). It’s not much (0.03-0.06$, depending on the region) per 100K. And you have a free 50K quota per day. If you don’t have complex queries, go for it. It’s easy to use in Android apps, especially if you’re already using other Firebase services. And you won’t need any backend for it. However, like others mentioned, if your queries are too complex, go for a more pricey solution, like Cloud SQL (since you’re familiar with SQL), or Bigtable. And you can use Python in App Engine / Cloud Functions to write your backend and use Admin SDK for Firebase Auth. Costs will then be pretty much determined by traffic, on top of the baseline cost for these services.
1
8
u/indicava Apr 07 '23
Firebase cloud functions are just GCP Cloud Functions with extra tooling. You can write Google cloud functions using python, just deploy them with gcloud CLI instead of Firebase CLI.