Storing Images in S3 from Node Server

Learn how to store your web app's files in an s3 bucket. Upload, Download, Update, and Delete images in an s3 bucket, from a node.js application.

Uploading an image goes through the express server allowing us to modify the image before it's stored in the s3 bucket. Downloading the image happens directly from the s3 bucket to put less strain on the server and make it easier to integrate our bucket with a CDN in the future.

• 0:00​ Intro
• 3:17 Post a photo with multipart/form-data
• 5:06 Multer
• 8:40 Create an S3 Bucket
• 11:38 IAM User and Policy
• 16:36 AWS SDK S3 Client
• 19:00 Uploading an image to S3
• 22:07 Updating an image
• 23:18 Random Image Names
• 25:16 Resizing Images
• 27:36 Saving data to the database
• 29:55 Getting images with signed url
• 35:28 Deleting an image
• 38:19 Summary

Examples of how to upload, download and delete files from S3 bucket using multer and the AWS SDK for JavaScript v3 Node.js.

https://github.com/meech-ward/s3-get-put-and-delete

Libraries:

• Multer: https://www.npmjs.com/package/multer
• AWS S3 Client: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html
• S3 request presigner: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_s3_request_presigner.html

To post an image to the server, the client sends a multipart/form-data request to the server with the image data and any other data that the client wants to send.

HTML:

<form action="/posts" method="POST" enctype="multipart/form-data">
   <input type="file" name="image" accept="image/*"/>
   <input type="text" name="caption" placeholder="Caption"/>
   <button type="submit">Submit</button>
</form>

React/Next:

export default function NewPost() {  
  const [file, setFile] = useState()
  const [caption, setCaption] = useState("")

  const submit = async event => {
    event.preventDefault()

    const formData = new FormData();
    formData.append("image", file)
    formData.append("caption", caption)
    await axios.post("/api/posts", formData, { headers: {'Content-Type': 'multipart/form-data'}})
  }

  return (
     <form onSubmit={submit}>
       <input onChange={e => setFile(e.target.files[0])} type="file" accept="image/*"></input>
       <input value={caption} onChange={e => setCaption(e.target.value)} type="text" placeholder='Caption'></input>
       <button type="submit">Submit</button>
     </form>
  )
}

The server then accepts the image data using multer and keeps the image in memory so it can be easily modified and sent to S3. The app's in this example modify the image by resizing it using sharp. At all times the image is stored in memory as a buffer.

import multer from 'multer'
import sharp from 'sharp'

const storage = multer.memoryStorage()
const upload = multer({ storage: storage })

app.post('/posts', upload.single('image'), async (req, res) => {
  const file = req.file 
  const caption = req.body.caption

  const fileBuffer = await sharp(file.buffer)
    .resize({ height: 1920, width: 1080, fit: "contain" })
    .toBuffer()

  // ...
})

Next we need to send the image to S3. First S3 needs to be configured with the correct credentials.

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3"

import dotenv from 'dotenv'

dotenv.config()

const bucketName = process.env.AWS_BUCKET_NAME
const region = process.env.AWS_BUCKET_REGION
const accessKeyId = process.env.AWS_ACCESS_KEY
const secretAccessKey = process.env.AWS_SECRET_ACCESS_KEY

const s3Client = new S3Client({
  region,
  credentials: {
    accessKeyId,
    secretAccessKey
  }
})

Then the image buffer data can be sent to S3 using the PutObjectCommand. Since the images in S3 all need to have unique names, we can use the crypto library to generate a unique unguessable name.

import crypto from 'crypto'

const generateFileName = (bytes = 32) => crypto.randomBytes(bytes).toString('hex')

app.post('/posts', upload.single('image'), async (req, res) => {
  const file = req.file 
  const caption = req.body.caption

  const fileBuffer = await sharp(file.buffer)
    .resize({ height: 1920, width: 1080, fit: "contain" })
    .toBuffer()

  // Configure the upload details to send to S3
  const fileName = generateFileName()
  const uploadParams = {
    Bucket: bucketName,
    Body: fileBuffer,
    Key: fileName,
    ContentType: file.mimetype
  }

  // Send the upload to S3
  await s3Client.send(new PutObjectCommand(uploadParams));

  // Save the image name to the database. Any other req.body data can be saved here too but we don't need any other image data.
  const post = await prisma.posts.create({
    data: {
      imageName,
      caption,
    }
  })

  res.send(post)
})

Once the images are being successfully uploaded to S3, we need to generate a signed URL so the client can GET the image from S3. The database only stores the image name, so we generate a signed URL using the image name.

import { GetObjectCommand } from "@aws-sdk/client-s3"
import { getSignedUrl } from "@aws-sdk/s3-request-presigner"

app.get("/", async (req, res) => {
  const posts = await prisma.posts.findMany({ orderBy: [{ created: 'desc' }] }) // Get all posts from the database

  for (let post of posts) { // For each post, generate a signed URL and save it to the post object
    post.imageUrl = await getSignedUrl(
      s3Client,
      new GetObjectCommand({
        Bucket: bucketName,
        Key: imageName
      }),
      { expiresIn: 60 }// 60 seconds
    )
  }

  res.send(posts)
})

https://aws.amazon.com/blogs/developer/generate-presigned-url-modular-aws-sdk-javascript/

Then any client can GET the image from S3 using the signed URL in the src of an img tag.

If you want to delete the image from S3, you can use the DeleteObjectCommand passing in the image name, then delete the corresponding post from the database.

app.delete("/api/posts/:id", async (req, res) => {
  const id = +req.params.id
  const post = await prisma.posts.findUnique({where: {id}}) 

  const deleteParams = {
    Bucket: bucketName,
    Key: post.imageName,
  }

  return s3Client.send(new DeleteObjectCommand(deleteParams))

  await prisma.posts.delete({where: {id}})
  res.send(post)
})