Search for a command to run...
I am a Data Engineer I love Python! I am based in Singapore I have good knowledge on cloud I have some knowledge on building Machine Learning Models I have helped 30+ companies use data effectively I have worked with Petabytes scale of data I write @ TowardsDataScience Nowadays I am focusing on learning Natural Language Processing Ask me about anything, I'll be happy to help.
No comments yet. Be the first to comment.
Great Expectations Tutorial
Data Engineering vs. Data Science: Choosing Your Data Path
Future-Proof Your Success: Unveiling the Must-Have Certifications for 2024 and Beyond!
In today's fast-paced digital world, keeping up with the latest advancements in data engineering is crucial to stay ahead of the competition. With the amount of data collected every day increasing, data engineering plays an important role in guarante...
Data Engineering Fundamentals with Python
To The Cloud and Beyond! I got you, fam!
 on Unsplash*](https://cdn-images-1.medium.com/max/2000/1*HSSsJ_6lWyyYDsRb-wsevA.png)
[Tanner Boriack](https://unsplash.com/@tannerboriack?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on Unsplash**
Choosing the right GCP Database depends on a lot of factors including your workload and the architecture involved. Today, I’m going to provide you all with an overview of popular Google cloud database services, including key considerations when assessing and choosing a service.
Google Cloud Platform (GCP) was built to provide an array of computing resources, database services being one of them. Competent and capable of handling modern data, bound with efficiency, flexibility, and great performance, GCP is a hosted platform solution for disseminated data across geography.
When choosing a Google database service, one should consider a lot of things like type and size of data, latency, throughput, scalability, and IOPs, to name a few.
1) Single — The simplest of all deployment models, one can deploy databases by creating new cloud databases on Google and/or by ‘lift and shift’ of pre-existing workloads.
2) Hybrid — These types of deployments are useful when one has applications in the cloud that need to access on-premises databases or vice-versa.
There are three primary factors to be considered when deploying a hybrid model (with some data on Google Cloud and some on-premises) :
Master Database: First and foremost you need to decide whether your master database is stored on-premises or on the cloud. Once you choose the cloud, GCP resources act as a data hub for on-premises resources, whereas if you choose on-premises, your in-house resources sync data to the cloud for remote use or backup.
Managed Services: Available for resources in the cloud, these services comprise scalability, redundancy, and automated backups. You, however, have an option of using third-party managed services.
Portability: Based on the type of data store you choose, the portability of your data is affected too. To ensure reliable and consistent transfer of data, you need to consider a cross-platform store, such as MySQL.
3) Multicloud — These types of deployments can help you effectively distribute your database and create multiple fail-safes as it enables you to combine databases deployed on Google Cloud with database services from other cloud providers thereby giving you an advantage of a wider array of proprietary cloud features.
There are 2 primary factors to be considered when deploying this model:
Integration: Ensuring that client systems can seamlessly access databases, regardless of the cloud they are deployed on, for instance, use of open-source client libraries to make databases smoothly available across clouds.
Migration: Since there are multiple cloud providers, one may need to migrate data between clouds with the help of database replication tools or export/import processes. Google Storage Transfer service is one such tool to help you with database migration.
GCP offers several database services that you can choose from.
A relational GCP database service that is fully managed and compatible with MySQL, PostgreSQL and SQL Server, Cloud SQL includes features like automated backups, data replication, and disaster recovery to ensure high availability and flexibility.
When to choose: From ‘lift and shift’ of on-premise SQL databases to the cloud to handling large-scale SQL data analytics to supporting CMS data storage and scalability and deployment of micro services, Cloud SQL has many uses and is a better option when you need relational database capabilities but don’t need storage capacity over 10TB.
Another fully managed, relational Google Cloud database service, Cloud Spanner differs from Cloud SQL by focusing on combining the benefits of relational structure and non-relational scalability. It provides consistency across rows and high-performance operations and includes features like built-in security, automatic replication, and multi-language support.
When to choose: Cloud Spanner should be your go-to option if you plan on using large amounts of data (more than 10TB) and need transactional consistency. It is also a perfect choice if you wish to use sharding for higher throughput and accessibility.
With BigQuery you can perform data analyses via SQL and query stream-data. Since BigQuery is a serverless data warehouse that’s fully managed, its built-in Data Transfer Service helps you migrate data from on-premises resources, including Teradata.
It incorporates features for machine learning, business intelligence, and geospatial analysis that are provided through BigQuery ML, BI Engine, and GIS.
When to choose: Use cases for BigQuery involve process analytics and optimization, big data processing and analytics, data warehouse modernisation, machine learning-based behavioural analytics and predictions.
It is a fully managed NoSQL Google Cloud database service that is designed for large operational and analytics workloads. Cloud Bigtable includes features for high availability and zero-downtime configuration changes. You can practically integrate it with a variety of tools, including Apache tools and Google Cloud services.
Cloud Bigtable use cases cover financial analysis and prediction, IoT data ingestion, processing, and analytics, and hyper-personalised marketing applications.
When to choose: Cloud Bigtable is a good option if you are using large amounts of single key data and is preferable for low-latency, high throughput workloads.
A fully managed, serverless NoSQL GCP database designed for the development of serverless apps, Cloud Firestore can be used to store, sync, and query data for web, mobile, and IoT applications. With critical features like offline support, live synchronization, and built-in security, you can even integrate Firestore with Firebase, GCP’s mobile development platform, for easier app creation.
Cloud Firestore use cases include mobile and web applications with both online and offline capabilities, multi-user, collaborative applications, real-time analytics, social media applications, and gaming forums and leaderboards.
When to choose: When your focus lies on app development and you need live synchronization and offline support.
This is a NoSQL Google Cloud database that is a part of the Firebase platform. It allows you to store and sync data in real-time and includes caching capabilities for offline use. It also enables you to implement declarative authentication, matching users by identity or pattern.
It includes mobile and web software development kits for easier app development. Use cases for Firebase Realtime Database involve development of apps that work across devices, advertisement optimisation and personalisation, and third-party payment processing.
Designed to be secure, highly available, and scalable, Cloud Memorystore is a fully managed, in-memory Google Cloud data store that enables you to create application caches with sub-millisecond latency for data access.
Use cases for Cloud Memorystore include ‘lift and shift’ migration of applications, machine learning applications, real-time analytics, low latency data caching and retrieval.
When to choose: If you are using key-value datasets and your main focus is transaction latency.
I also created this flowchart that can show a direction in terms of selecting the database:
](https://cdn-images-1.medium.com/max/8840/1*uN6DQstlLn9YT_ZbPhS8IQ.png)
Created by Anuj Syal, whimsical link
I created this video on youtube that covers this topic in detail
With more and more enterprises looking to the cloud services to run their core-critical systems and also act as a primary platform for launching their new apps, choosing the right Google Cloud Platform (GCP) Database opens possibilities and can be a dealmaker, offering the least downtime for customers, secure and cost-effective services and solutions.
If you are interested in similar content hit up the follow button on Medium or follow me on Twitter and Linkedin