Storage Emulator Hangs with Parallel Uploads (4+ Concurrent Requests)

[luckyape]

[REQUIRED] Environment info

• firebase-tools: 13.4.1
• Platform: macOS Sonoma 14.3.1

[REQUIRED] Test case

      const media = [jpg1, jpg2, jpg3, jpg4]; // files range in size from 1mb to 1.5mb
      const uploadPromises = media.map((file) => {

        const storageRef = ref(storage, `temp/${file.path}`);
        return await uploadBytes(storageRef, file);
      });
      await Promise.all(uploadPromises);
      console.log('Upload promises complete');

[REQUIRED] Steps to reproduce

Run code with firebase storage and emulators configured. Observe that only the first two images upload successfully.

[REQUIRED] Expected behavior

4 images should be uploaded to storage.
Promises should resolve and 'Upload promises complete' should be logged to the console.
Synchronous execution of the uploads does work.

[REQUIRED] Actual behavior

Only 2 images are uploaded to storage.
Promises hang and do not resolve.

[debug] [2024-03-10T20:18:06.822Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:06.854Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:06.856Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:06.918Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:07.159Z] Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:07.216Z] Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[info] {"result":{"permit":true},"id":15,"status":"ok"}
{"result":{"permit":true},"id":16,"status":"ok"} {"metadata":{"emulator":{"name":"storage"},"message":"{"result":{"permit":true},"id":15,"status":"ok"}\n{"result":{"permit":true},"id":16,"status":"ok"}"}}
[debug] [2024-03-10T20:18:07.947Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none]
[debug] [2024-03-10T20:18:07.947Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"eventId":"1710101887947","timestamp":"2024-03-10T20:18:07.947Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","type":"storage#object"},"data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","bucket":"hellograham.appspot.com","generation":"1710101887908","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.908Z","updated":"2024-03-10T20:18:07.908Z","storageClass":"STANDARD","size":"1133688","md5Hash":"6XXm9KGgFk9ge8Pl/YYnpw==","etag":"6TEW5Thl0dy3G0IbWd9RgR8D3iA","metadata":{"firebaseStorageDownloadTokens":"906fda07-0dd7-4c49-85b5-0edb4e36d9b4"},"crc32c":"igflCw==","timeStorageClassUpdated":"2024-03-10T20:18:07.908Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg/1710101887908","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg?generation=1710101887908&alt=media"}}
[debug] [2024-03-10T20:18:07.951Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200
[debug] [2024-03-10T20:18:07.951Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"}
[debug] [2024-03-10T20:18:07.952Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none]
[debug] [2024-03-10T20:18:07.952Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"specversion":"1.0","id":"338f9c11-a6ab-4b3e-9895-b073b28fcbb2","type":"google.cloud.storage.object.v1.finalized","source":"//storage.googleapis.com/projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","time":"2024-03-10T20:18:07.908Z","data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","bucket":"hellograham.appspot.com","generation":"1710101887908","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.908Z","updated":"2024-03-10T20:18:07.908Z","storageClass":"STANDARD","size":"1133688","md5Hash":"6XXm9KGgFk9ge8Pl/YYnpw==","etag":"6TEW5Thl0dy3G0IbWd9RgR8D3iA","metadata":{"firebaseStorageDownloadTokens":"906fda07-0dd7-4c49-85b5-0edb4e36d9b4"},"crc32c":"igflCw==","timeStorageClassUpdated":"2024-03-10T20:18:07.908Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg/1710101887908","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg?generation=1710101887908&alt=media"}}
[debug] [2024-03-10T20:18:07.955Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200
[debug] [2024-03-10T20:18:07.955Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"}
[debug] [2024-03-10T20:18:07.959Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none]
[debug] [2024-03-10T20:18:07.959Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"eventId":"1710101887959","timestamp":"2024-03-10T20:18:07.959Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","type":"storage#object"},"data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","bucket":"hellograham.appspot.com","generation":"1710101887923","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.923Z","updated":"2024-03-10T20:18:07.923Z","storageClass":"STANDARD","size":"1515111","md5Hash":"KyqvqiFDwGtMrSnrGyXyYg==","etag":"y2qK0V857ty6e0G93oORBd4dtlc","metadata":{"firebaseStorageDownloadTokens":"a5c9469a-acd9-4e37-8115-bcca4c0028ed"},"crc32c":"w2JVsA==","timeStorageClassUpdated":"2024-03-10T20:18:07.923Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg/1710101887923","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg?generation=1710101887923&alt=media"}}
[debug] [2024-03-10T20:18:07.963Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200
[debug] [2024-03-10T20:18:07.963Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"}
[debug] [2024-03-10T20:18:07.963Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none]
[debug] [2024-03-10T20:18:07.963Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"specversion":"1.0","id":"9692e1ed-b450-4b40-a74f-4e1db1dbeae7","type":"google.cloud.storage.object.v1.finalized","source":"//storage.googleapis.com/projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","time":"2024-03-10T20:18:07.923Z","data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","bucket":"hellograham.appspot.com","generation":"1710101887923","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.923Z","updated":"2024-03-10T20:18:07.923Z","storageClass":"STANDARD","size":"1515111","md5Hash":"KyqvqiFDwGtMrSnrGyXyYg==","etag":"y2qK0V857ty6e0G93oORBd4dtlc","metadata":{"firebaseStorageDownloadTokens":"a5c9469a-acd9-4e37-8115-bcca4c0028ed"},"crc32c":"w2JVsA==","timeStorageClassUpdated":"2024-03-10T20:18:07.923Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg/1710101887923","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg?generation=1710101887923&alt=media"}}
[debug] [2024-03-10T20:18:07.965Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200
[debug] [2024-03-10T20:18:07.965Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"}

[aalej]

Hey @luckyape, thanks for reaching out and for providing a detailed report and description of the issue you’re encountering. I’m currently working on replicating this, but I’m unable to reproduce the same behavior. When uploading 4 files (~1MB in size each) at the same time, all of them get uploaded without any issues to the Storage emulator.

I created this repo to try and replicate this. Could you let me know if you’re able to reproduce the issue using the steps provided, or if I’m missing anything to reproduce the issue? Also, if you can provide a minimal reproducible example, please share it so that we can replicate the issue on our end.

[luckyape]

@aalej Thanks again for your help. I have been able to recreate the error in the React Admin context. https://github.com/luckyape/issue-6865-react-admin

…

On Wed, Mar 13, 2024 at 6:18 AM graham charles ***@***.***> wrote: for now I take it back...seems there is still something going on, looping back on trying to isolate the issue On Tue, Mar 12, 2024 at 7:05 PM graham charles ***@***.***> wrote: > Thanks for the quick response. The error was on my end, as I was > returning an `await storagePromise` rather than just the promise. > > If you would like to see how to produce the error behavior for some > reason: https://github.com/luckyape/issue-6865 > > I will update the ticket incase anyone else is trying to debug the same > issue. > > graham > > On Tue, Mar 12, 2024 at 3:26 PM aalej ***@***.***> wrote: > >> Hey @luckyape <https://github.com/luckyape>, thanks for reaching out >> and for providing a detailed report and description of the issue you’re >> encountering. I’m currently working on replicating this, but I’m unable to >> reproduce the same behavior. When uploading 4 files (~1MB in size each) at >> the same time, all of them get uploaded without any issues to the Storage >> emulator. >> >> I created this repo <https://github.com/aalej/issues-6865> to try and >> replicate this. Could you let me know if you’re able to reproduce the issue >> using the steps provided, or if I’m missing anything to reproduce the >> issue? Also, if you can provide a minimal reproducible example >> <https://stackoverflow.com/help/minimal-reproducible-example>, please >> share it so that we can replicate the issue on our end. >> >> — >> Reply to this email directly, view it on GitHub >> <#6865 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AAGB2UEYY4XQIEU3JSGU2TTYX5CFPAVCNFSM6AAAAABEPH4N5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJSGI4TGMBXHA> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> > > > -- > Please lmk if you have any questions. > g > -- Please lmk if you have any questions. g

-- Please lmk if you have any questions. g

[aalej]

Thanks @luckyape for sharing a repo for this. I tried using the application you shared, but I’m still unable to reproduce the issue. The steps I took were:

1. Run firebase emulators:start --project project_id
2. Run yarn to install dependencies
3. Add Firebase configs to https://github.com/luckyape/issue-6865-react-admin/blob/6a9695dc58ed9c4bf55ea29b10dbb6084cf4511d/src/App.tsx#L7-L9
4. Run yarn dev to start the app
5. Open "http://localhost:5173/#/firebase/create"
6. Select files to upload
7. Click the “UPLOAD” button
8. Checking the Storage emulator shows that the files were uploaded:

I think I’m probably missing something here to reproduce the issue. To make sure that we're on the same page, could you let me know if the steps I took to reproduce the issue are correct?

[luckyape]

i have some other tasks prioritized at the moment, but i will quickly add for now that the behavior does upload some of the images before failing. ensure the console is clean and perhaps test with additional images. I promise to loop back and try to get a more detailed response by the beginning of next week. thanks again for your help @aalej

…

On Fri, Mar 15, 2024 at 10:39 AM aalej ***@***.***> wrote: Thanks @luckyape <https://github.com/luckyape> for sharing a repo for this. I tried using the application you shared, but I’m still unable to reproduce the issue. The steps I took were: 1. Run firebase emulators:start --project project_id 2. Run yarn to install dependencies 3. Add Firebase configs to https://github.com/luckyape/issue-6865-react-admin/blob/6a9695dc58ed9c4bf55ea29b10dbb6084cf4511d/src/App.tsx#L7-L9 4. Run yarn dev to start the app 5. Open "http://localhost:5173/#/firebase/create" 6. Select files to upload 7. Click the “UPLOAD” button 8. Checking the Storage emulator shows that the files were uploaded: storage-em-ss.png (view on web) <https://github.com/firebase/firebase-tools/assets/118378103/7456217b-afe6-4f0c-b955-0ad3d37f96a4> I think I’m probably missing something here to reproduce the issue. To make sure that we're on the same page, could you let me know if the steps I took to reproduce the issue are correct? — Reply to this email directly, view it on GitHub <#6865 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGB2UD4FOSA6F7XLZK5AODYYL2ZZAVCNFSM6AAAAABEPH4N5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJZGY4DONZWGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Please lmk if you have any questions. g