Move File From /tmp Folder To Google Cloud Storage Bucket
I originally posted this question when I was having trouble getting my python cloud function to create and write to a new file. Since then I've managed to create a csv in the /tmp
Solution 1:
to move that file into my bucket
Here is an example. Bear in mind:
- Don't copy and paste without thinking.
- The code snippet is only to show the idea - it won't work as is. Modifications are required to fit into your context and requirements.
- The
_crc32sum
function was not developed by me. - I did not test the code. It is just from my head with copying some elements from different public sources.
Here is the code:
import base64
import crc32c
import os
from google.cloud import exceptions
from google.cloud import storage
# =====> ==============================# a function to calculate crc32c hash
def _crc32sum(filename: str, blocksize: int = 65536) -> int:
"""Calculate the crc32c hash for a file with the provided name
:param filename: the name of the file
:param blocksize: the size of the block for the file reading
:return: the calculated crc32c hash for the given file
"""
checksum = 0
with open(filename, "rb") as f_ref:
for block in iter(lambda: f_ref.read(blocksize), b""):
checksum = crc32c.crc32(block, checksum)
return checksum & 0xffffffff# =====> ==============================# use the default project in the client initialisation
CS = storage.Client()
lcl_file_name = "/tmp/my-local-file.csv"
tgt_bucket_name = "my-bucket-name"
tgt_object_name = "prefix/another-prefix/my-target-file.csv"# =====> ==============================# =====> ==============================# =====> the process strats here# https://googleapis.dev/python/storage/latest/_modules/google/cloud/storage/client.html#Client.lookup_bucket
gcs_tgt_bucket_ref = CS.lookup_bucket(tgt_bucket_name)
# check if the target bucket does existif gcs_tgt_bucket_ref is None:
# handle incorrect bucket name or its absence# most likely we are to finish the execution here rather than 'pass'
pass
# calculate the hash for the local file
lcl_crc32c = _crc32sum(lcl_file_name)
base64_crc32c = base64.b64encode(lcl_crc32c.to_bytes(
length=4, byteorder='big')).decode('utf-8')
# check if the file/object in the bucket already exists# https://googleapis.dev/python/storage/latest/_modules/google/cloud/storage/bucket.html#Bucket.blob
gcs_file_ref = gcs_tgt_bucket_ref.blob(tgt_object_name)
# https://googleapis.dev/python/storage/latest/_modules/google/cloud/storage/blob.html#Blob.existsif gcs_file_ref.exists():
gcs_file_ref.reload()
# compare crc32c hashes - between the local file and the gcs file/objectif base64_crc32c != gcs_file_ref.crc32c:
# the blob file/object in the GCS has a different hash# the blob file/object should be deleted and a new one to be uploaded# https://googleapis.dev/python/storage/latest/_modules/google/cloud/storage/blob.html#Blob.delete
gcs_file_ref.delete()
else:
# the file/object is already in the bucket# most likely we are to finish the execution here rather than 'pass'
pass
# upload file to the target bucket# reinit the reference in case the target file/object was deleted
gcs_file_ref = gcs_tgt_bucket_ref.blob(tgt_file_name)
gcs_file_ref.crc32c = base64_crc32c
with open(lcl_file_name, 'rb') as file_obj:
try:
gcs_file_ref.metadata = {
"custom-metadata-key": "custom-metadata-value"
}
# https://googleapis.dev/python/storage/latest/_modules/google/cloud/storage/blob.html#Blob.upload_from_file
gcs_file_ref.upload_from_file(
file_obj=file_obj, content_type="text/csv", checksum="crc32c")
except exceptions.GoogleCloudError as gc_err:
# handle the exception here# don't forget to delete the local file if it is not required anymore# most likely we are to finish the execution here rather than 'pass'
pass
# clean behindif lcl_file_name and os.path.exists(lcl_file_name):
os.remove(lcl_file_name)
# =====> the process ends here# =====> ==============================
Let me know if there are significant mistakes, and I modify the example.
Post a Comment for "Move File From /tmp Folder To Google Cloud Storage Bucket"