Cloud Storage Tutorial

Preparing the application

1. Create a Cloud Storage bucket to upload a test file, where YOUR_TRIGGER_BUCKET_NAME is a globally unique bucket name:

   gsutil mb gs://YOUR_TRIGGER_BUCKET_NAME
   

2. Clone the sample app repository to your local machine:

   Node.js 6

   git clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

   Node.js 8

   git clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

   Python

   git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

   Go

   git clone https://github.com/GoogleCloudPlatform/golang-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

3. Change to the directory that contains the Cloud Functions sample code:

   Node.js 6

   cd nodejs-docs-samples/functions/helloworld/

   Node.js 8

   cd nodejs-docs-samples/functions/node8/

   Python

   cd python-docs-samples/functions/gcs/

   Go

   cd golang-samples/functions/helloworld/storage_generic/

Deploying and triggering the function

Currently, Cloud Storage functions are based on Pub/Sub notifications from Cloud Storage and support similar event types:

• finalize

• delete

• archive

• metadata update

The following sections describe how to deploy and trigger a function for each of these event types.

Object Finalize

Object finalize events trigger when a "write" of a Cloud Storage Object is successfully finalized. In particular, this means that creating a new object or overwriting an existing object triggers this event. Archive and metadata update operations are ignored by this trigger.

Object Finalize: deploying the function

Take a look at the sample function, which handles Cloud Storage events:

/**
 * Generic background Cloud Function to be triggered by Cloud Storage.
 *
 * @param {object} event The Cloud Functions event.
 * @param {function} callback The callback function.
 */
exports.helloGCSGeneric = (event, callback) => {
  const file = event.data;

  console.log(`  Event: ${event.eventId}`);
  console.log(`  Event Type: ${event.eventType}`);
  console.log(`  Bucket: ${file.bucket}`);
  console.log(`  File: ${file.name}`);
  console.log(`  Metageneration: ${file.metageneration}`);
  console.log(`  Created: ${file.timeCreated}`);
  console.log(`  Updated: ${file.updated}`);

  callback();
};

/**
 * Generic background Cloud Function to be triggered by Cloud Storage.
 *
 * @param {object} data The event payload.
 * @param {object} context The event metadata.
 */
exports.helloGCSGeneric = (data, context) => {
  const file = data;
  console.log(`  Event ${context.eventId}`);
  console.log(`  Event Type: ${context.eventType}`);
  console.log(`  Bucket: ${file.bucket}`);
  console.log(`  File: ${file.name}`);
  console.log(`  Metageneration: ${file.metageneration}`);
  console.log(`  Created: ${file.timeCreated}`);
  console.log(`  Updated: ${file.updated}`);
};

def hello_gcs_generic(data, context):
    """Background Cloud Function to be triggered by Cloud Storage.
       This generic function logs relevant data when a file is changed.

    Args:
        data (dict): The Cloud Functions event payload.
        context (google.cloud.functions.Context): Metadata of triggering event.
    Returns:
        None; the output is written to Stackdriver Logging
    """

    print('Event ID: {}'.format(context.event_id))
    print('Event type: {}'.format(context.event_type))
    print('Bucket: {}'.format(data['bucket']))
    print('File: {}'.format(data['name']))
    print('Metageneration: {}'.format(data['metageneration']))
    print('Created: {}'.format(data['timeCreated']))
    print('Updated: {}'.format(data['updated']))

// Package helloworld provides a set of Cloud Functions samples.
package helloworld

import (
	"context"
	"fmt"
	"log"
	"time"

	"cloud.google.com/go/functions/metadata"
)

// GCSEvent is the payload of a GCS event. Please refer to the docs for
// additional information regarding GCS events.
type GCSEvent struct {
	Bucket         string    `json:"bucket"`
	Name           string    `json:"name"`
	Metageneration string    `json:"metageneration"`
	ResourceState  string    `json:"resourceState"`
	TimeCreated    time.Time `json:"timeCreated"`
	Updated        time.Time `json:"updated"`
}

// HelloGCSInfo prints information about a GCS event.
func HelloGCSInfo(ctx context.Context, e GCSEvent) error {
	meta, err := metadata.FromContext(ctx)
	if err != nil {
		return fmt.Errorf("metadata.FromContext: %v", err)
	}
	log.Printf("Event ID: %v\n", meta.EventID)
	log.Printf("Event type: %v\n", meta.EventType)
	log.Printf("Bucket: %v\n", e.Bucket)
	log.Printf("File: %v\n", e.Name)
	log.Printf("Metageneration: %v\n", e.Metageneration)
	log.Printf("Created: %v\n", e.TimeCreated)
	log.Printf("Updated: %v\n", e.Updated)
	return nil
}

To deploy the function, run the following command in the directory where the sample code is located:

gcloud functions deploy helloGCSGeneric --runtime nodejs6 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.finalize

gcloud functions deploy helloGCSGeneric --runtime nodejs8 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.finalize

gcloud functions deploy hello_gcs_generic --runtime python37 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.finalize

gcloud functions deploy HelloGCSInfo --runtime go111 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.finalize

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage bucket that triggers the function.

Object Finalize: triggering the function

To trigger the function:

1. Create an empty gcf-test.txt file in the directory where the sample code is located.

2. Upload the file to Cloud Storage in order to trigger the function:

   gsutil cp gcf-test.txt gs://YOUR_TRIGGER_BUCKET_NAME
   

   where YOUR_TRIGGER_BUCKET_NAME is the name of your Cloud Storage bucket where you will upload a test file.

3. Check the logs to make sure the executions have completed:

   gcloud functions logs read --limit 50
   

Object Delete

Object delete events are most useful for non-versioning buckets. They are triggered when an old version of an object is deleted. In addition, they are triggered when an object is overwritten. Object delete triggers can also be used with versioning buckets, triggering when a version of an object is permanently deleted.

Object Delete: deploying the function

Using the same sample code as in the finalize example, deploy the function with object delete as the trigger event. Run the following command in the directory where the sample code is located:

gcloud functions deploy helloGCSGeneric --runtime nodejs6 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.delete

gcloud functions deploy helloGCSGeneric --runtime nodejs8 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.delete

gcloud functions deploy hello_gcs_generic --runtime python37 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.delete

gcloud functions deploy HelloGCSInfo --runtime go111 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.delete

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage bucket that triggers the function.

Object Delete: triggering the function

To trigger the function:

1. Create an empty gcf-test.txt file in the directory where the sample code is located.

2. Make sure that your bucket is non-versioning:

   gsutil versioning set off gs://YOUR_TRIGGER_BUCKET_NAME
   

3. Upload the file to Cloud Storage:

   gsutil cp gcf-test.txt gs://YOUR_TRIGGER_BUCKET_NAME
   

   where YOUR_TRIGGER_BUCKET_NAME is the name of your Cloud Storage bucket where you will upload a test file. At this point the function should not execute yet.

4. Delete the file to trigger the function:

   gsutil rm gs://YOUR_TRIGGER_BUCKET_NAME/gcf-test.txt
   

5. Check the logs to make sure the executions have completed:

   gcloud functions logs read --limit 50
   

Note that the function may take some time to finish executing.

Object Archive

Object archive events can be used only with versioning buckets. They are triggered when an old version of an object is archived. In particular, this means that when an object is overwritten or deleted, an archive event is triggered.

Object Archive: deploying the function

Using the same sample code as in the finalize example, deploy the function with object archive as the trigger event. Run the following command in the directory where the sample code is located:

gcloud functions deploy helloGCSGeneric --runtime nodejs6 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.archive

gcloud functions deploy helloGCSGeneric --runtime nodejs8 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.archive

gcloud functions deploy hello_gcs_generic --runtime python37 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.archive

gcloud functions deploy HelloGCSInfo --runtime go111 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.archive

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage bucket that triggers the function.

Object Archive: triggering the function

To trigger the function:

1. Create an empty gcf-test.txt file in the directory where the sample code is located.

2. Make sure that your bucket has versioning enabled:

   gsutil versioning set on gs://YOUR_TRIGGER_BUCKET_NAME
   

3. Upload the file to Cloud Storage:

   gsutil cp gcf-test.txt gs://YOUR_TRIGGER_BUCKET_NAME
   

   where YOUR_TRIGGER_BUCKET_NAME is the name of your Cloud Storage bucket where you will upload a test file. At this point the function should not execute yet.

4. Archive the file to trigger the function:

   gsutil rm gs://YOUR_TRIGGER_BUCKET_NAME/gcf-test.txt
   

5. Watch the logs to make sure the executions have completed:

   gcloud functions logs read --limit 50
   

Object Metadata Update

Metadata update events are triggered when the metadata of existing object is updated.

Object Metadata Update: deploying the function

Using the same sample code as in the finalize example, deploy the function with metadata update as the trigger event. Run the following command in the directory where the sample code is located:

gcloud functions deploy helloGCSGeneric --runtime nodejs6 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.metadataUpdate

gcloud functions deploy helloGCSGeneric --runtime nodejs8 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.metadataUpdate

gcloud functions deploy hello_gcs_generic --runtime python37 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.metadataUpdate

gcloud functions deploy HelloGCSInfo --runtime go111 --trigger-resource YOUR_TRIGGER_BUCKET_NAME --trigger-event google.storage.object.metadataUpdate

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage bucket that triggers the function.

Object Metadata Update: triggering the function

To trigger the function:

1. Create an empty gcf-test.txt file in the directory where the sample code is located.

2. Make sure that your bucket is non-versioning:

   gsutil versioning set off gs://YOUR_TRIGGER_BUCKET_NAME
   

3. Upload the file to Cloud Storage:

   gsutil cp gcf-test.txt gs://YOUR_TRIGGER_BUCKET_NAME
   

   where YOUR_TRIGGER_BUCKET_NAME is the name of your Cloud Storage bucket where you will upload a test file. At this point the function should not execute yet.

4. Update the metadata of the file:

   gsutil -m setmeta -h "Content-Type:text/plain" gs://YOUR_TRIGGER_BUCKET_NAME/gcf-test.txt
   

5. Watch the logs to make sure the executions have completed:

   gcloud functions logs read --limit 50