54

Currently, I am using the @google-cloud/storage NPM package to upload a file directly to a Google Cloud Storage bucket. This requires some trickery as I only have the image's base64 encoded string. I have to:

  • Decode the string
  • Save it as a file
  • Send the file path to the below script to upload to Google Cloud Storage
  • Delete the local file

I'd like to avoid storing the file in the filesystem altogether since I am using Google App Engine and I don't want to overload the filesystem / leave junk files there if the delete operation doesn't work for whatever reason. This is what my upload script looks like right now:

// Convert the base64 string back to an image to upload into the Google Cloud Storage bucket
var base64Img = require('base64-img');
var filePath = base64Img.imgSync(req.body.base64Image, 'user-uploads', 'image-name');

// Instantiate the GCP Storage instance
var gcs = require('@google-cloud/storage')(),
    bucket = gcs.bucket('google-cloud-storage-bucket-name');

// Upload the image to the bucket
bucket.upload(__dirname.slice(0, -15) + filePath, {
    destination: 'profile-images/576dba00c1346abe12fb502a-original.jpg',
    public: true,
    validation: 'md5'
}, function(error, file) {

    if (error) {
        sails.log.error(error);
    }

    return res.ok('Image uploaded');
});

Is there anyway to directly upload the base64 encoded string of the image instead of having to convert it to a file and then upload using the path?

Improve this question
2
  • bucket.upload wraps the file.createWriteStream function, so you'll need to pipe your base64 file string into the stream created by that method in file. I'd recommend just writing to the filesystem and unlinking after uploading though. I don't think you'll have problems with deleting the files. I might be able to pull off an example if you're so inclined. Commented Mar 18, 2017 at 22:07
  • @forrestmid Would really appreciate an example of how you would implement the file.createWriteStream to directly upload. Thanks! Commented Mar 19, 2017 at 8:19

7 Answers 7

Reset to default
53

The solution, I believe, is to use the file.createWriteStream functionality that the bucket.upload function wraps in the Google Cloud Node SDK.

I've got very little experience with streams, so try to bear with me if this doesn't work right off.

First of all, we need take the base64 data and drop it into a stream. For that, we're going to include the stream library, create a buffer from the base64 data, and add the buffer to the end of the stream.

var stream = require('stream');
var bufferStream = new stream.PassThrough();
bufferStream.end(Buffer.from(req.body.base64Image, 'base64'));

More on decoding base64 and creating the stream.

We're then going to pipe the stream into a write stream created by the file.createWriteStream function.

var gcs = require('@google-cloud/storage')({
  projectId: 'grape-spaceship-123',
  keyFilename: '/path/to/keyfile.json'
});

//Define bucket.
var myBucket = gcs.bucket('my-bucket');
//Define file & file name.
var file = myBucket.file('my-file.jpg');
//Pipe the 'bufferStream' into a 'file.createWriteStream' method.
bufferStream.pipe(file.createWriteStream({
    metadata: {
      contentType: 'image/jpeg',
      metadata: {
        custom: 'metadata'
      }
    },
    public: true,
    validation: "md5"
  }))
  .on('error', function(err) {})
  .on('finish', function() {
    // The file upload is complete.
  });

Info on file.createWriteStream, File docs, bucket.upload, and the bucket.upload method code in the Node SDK.

So the way the above code works is to define the bucket you want to put the file in, then define the file and the file name. We don't set upload options here. We then pipe the bufferStream variable we just created into the file.createWriteStream method we discussed before. In these options we define the metadata and other options you want to implement. It was very helpful to look directly at the Node code on Github to figure out how they break down the bucket.upload function, and recommend you do so as well. Finally, we attach a couple events for when the upload finishes and when it errors out.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Thank you for posting this! I actually did something similar except, I used the file.save() API which is a wraparound of createWriteStream.
@Nag That definitely works! I read through that API but didn't notice it's operation matched that which you were looking for. Glad you were able to figure it out.
@Nag how exactly did you manage to do that? do you have the source code we can take a look at?? I'm struggling a lot with this. I'm trying to upload a base64 encoded image string to Firebase Storage from the Firebase Cloud Functions
@krlozadan Please see my answer below. I'm not sure what the differences are between Cloud Storage and Firebase Storage so I cannot comment on Firebase. Hope this helps.
Note that the Buffer constructor is deprecated due to security issues. We should use Buffer.from(req.body.base64Image, 'base64') instead.
|
38

Posting my version of the answer in response to @krlozadan 's request above:

// Convert the base64 string back to an image to upload into the Google Cloud Storage bucket
var mimeTypes = require('mimetypes');

var image = req.body.profile.image,
    mimeType = image.match(/data:([a-zA-Z0-9]+\/[a-zA-Z0-9-.+]+).*,.*/)[1],
    fileName = req.profile.id + '-original.' + mimeTypes.detectExtension(mimeType),
    base64EncodedImageString = image.replace(/^data:image\/\w+;base64,/, ''),
    imageBuffer = new Buffer(base64EncodedImageString, 'base64');

// Instantiate the GCP Storage instance
var gcs = require('@google-cloud/storage')(),
    bucket = gcs.bucket('my-bucket');

// Upload the image to the bucket
var file = bucket.file('profile-images/' + fileName);

file.save(imageBuffer, {
    metadata: { contentType: mimeType },
    public: true,
    validation: 'md5'
}, function(error) {

    if (error) {
        return res.serverError('Unable to upload the image.');
    }

    return res.ok('Uploaded');
});

This worked just fine for me. Ignore some of the additional logic in the first few lines as they are only relevant to the application I am building.

Improve this answer

Comments

21

If you want to save a string as a file in Google Cloud Storage, you can do it easily using the file.save method:

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();
const myBucket = storage.bucket('my-bucket');

const file = myBucket.file('my-file.txt');
const contents = 'This is the contents of the file.';

file.save(contents).then(() => console.log('done'));
Improve this answer

3 Comments

The question is about uploading "base64 encoded image". This does not work.
This works for base64 strings if you set contents to this, where data is a base64 encoded file: Buffer.from(data.replace(/^data:image\/(png|gif|jpeg);base64,/, ''), 'base64');
I wanted to upload JSON string as a file, but it didn't work. Any help? This not even throwing any error.
7

:) what an issue !! Have tried it and got the issue Image has uploaded on firebase Storage but not download and just loader is moving around and around... After spending time... Got the success to upload the image on firebase storage with downloading... There was an issue in an access token... 

check the screenshot

enter image description here

If you check in the file location section on the right side bottom there is an option "create access token" and not showing any "access token" on there if you create manually access token on there then refresh the page image will showing... So now the question is how to create it by code...

just use below code to create the access token

const uuidv4 = require('uuid/v4');
const uuid = uuidv4();
metadata: { firebaseStorageDownloadTokens: uuid }

Full code is given below for uploading an image to storage image on firebase storage

const functions = require('firebase-functions')
var firebase = require('firebase');
var express = require('express');
var bodyParser = require("body-parser");

enter image description here

const uuidv4 = require('uuid/v4');
const uuid = uuidv4();

    const os = require('os')
    const path = require('path')
    const cors = require('cors')({ origin: true })
    const Busboy = require('busboy')
    const fs = require('fs')
    var admin = require("firebase-admin");


    var serviceAccount = {
        "type": "service_account",
        "project_id": "xxxxxx",
        "private_key_id": "xxxxxx",
        "private_key": "-----BEGIN PRIVATE KEY-----\jr5x+4AvctKLonBafg\nElTg3Cj7pAEbUfIO9I44zZ8=\n-----END PRIVATE KEY-----\n",
        "client_email": "[email protected]",
        "client_id": "xxxxxxxx",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/firebase-adminsdk-5rmdm%40xxxxx.iam.gserviceaccount.com"
      }

    admin.initializeApp({
        credential: admin.credential.cert(serviceAccount),
        storageBucket: "xxxxx-xxxx" // use your storage bucket name
    });


    const app = express();
    app.use(bodyParser.urlencoded({ extended: false }));
    app.use(bodyParser.json());
app.post('/uploadFile', (req, response) => {
    response.set('Access-Control-Allow-Origin', '*');
    const busboy = new Busboy({ headers: req.headers })
    let uploadData = null
    busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
        const filepath = path.join(os.tmpdir(), filename)
        uploadData = { file: filepath, type: mimetype }
        console.log("-------------->>",filepath)
        file.pipe(fs.createWriteStream(filepath))
      })

      busboy.on('finish', () => {
        const bucket = admin.storage().bucket();
        bucket.upload(uploadData.file, {
            uploadType: 'media',
            metadata: {
              metadata: { firebaseStorageDownloadTokens: uuid,
                contentType: uploadData.type,
              },
            },
          })

          .catch(err => {
            res.status(500).json({
              error: err,
            })
          })
      })
      busboy.end(req.rawBody)
   });




exports.widgets = functions.https.onRequest(app);
Improve this answer

1 Comment

Thanks for sharing your code! Could you also share how your request was formatted (what was the body of the POST request?)
5

You have to convert base64 to image buffer then upload as below, you need to provide image_data_from_html variable as the data you extract from HTML event.

const base64Text = image_data_from_html.split(';base64,').pop();
const imageBuffer = Buffer.from(base64Text, 'base64');
const contentType = data.image_data.split(';base64,')[0].split(':')[1];
const fileName = 'myimage.png';
const imageUrl = 'https://storage.googleapis.com/bucket-url/some_path/' + fileName;

await admin.storage().bucket().file('some_path/' + fileName).save(imageBuffer, {
    public: true,
    gzip: true,
    metadata: {
        contentType,
        cacheControl: 'public, max-age=31536000',
    }
});

console.log(imageUrl);
Improve this answer

1 Comment

It's worth pointing out that if your base64-encoded string is actually a dataURL (see: developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/…) like one you generated from a canvas or fabric, then you need this solution's first line to get rid of the extra URL parts. Works for me.
1

I was able to get the base64 string over to my Cloud Storage bucket with just one line of code.

var decodedImage = new Buffer(poster64, 'base64');
        

// Store Poster to storage
let posterFile = await client.file(decodedImage, `poster_${path}.jpeg`, { path: 'submissions/dev/', isBuffer: true, raw: true });
let posterUpload = await client.upload(posterFile, { metadata: { cacheControl: 'max-age=604800' }, public: true, overwrite: true });
let permalink = posterUpload.permalink

Something to be aware of is that if you are inside of a Nodejs environment you wont be able to use atob().

The top answer of this post showed me the errors of my ways! NodeJS base64 image encoding/decoding not quite working

Improve this answer

1 Comment

Not sure where you get isBuffer: true, raw: true from -- I don't see those in the SDK.
0 
const { Storage } = require('@google-cloud/storage');
const storage = new Storage() // start with your creadentials 
const contents = "data:image/jpeg..." // raw image data base64 encoded
const bucketName = "your google cloud bucket name";
const fileExtension = contents.split(";")[0].split("/")[1];
const destFileName = `${v4().toString()}.${fileExtension || "jpg"}` // random file name
const bucket = storage.bucket(bucketName);
// Create a reference to a file object
const file = bucket.file(destFileName || "filename.jpg");

// Create a pass through stream from a string
const passthroughStream = new stream.PassThrough();
passthroughStream.write(Buffer.from(contents.replace(/^data:image\/(png|gif|jpeg);base64,/, ''), 'base64'));
passthroughStream.end();

      async function streamFileUpload() {
        return new Promise((resolve, reject) => {
          passthroughStream.pipe(file.createWriteStream({
            gzip: true,
            metadata: {
              contentType,
              metadata: {
                custom: 'metadata'
              }
            }
          }))
            .on('finish', () => {
              resolve({
                url: `https://storage.googleapis.com/${bucketName}/${destFileName}`,
                title: destFileName,
              })
            }).on("error", (err) => {
              reject(err)
            });

          // console.log(result);
          console.log(`uploaded to '${bucketName}' as '${destFileName}'.`);
        })
      }

const result = await streamFileUpload().catch(console.error);

The result contains url and title of the file uploaded on google cloud

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.